2024-08-20T22:10:14.3401273Z Current runner version: '2.319.1' 2024-08-20T22:10:14.3407573Z Runner name: 'i-0dc72530ca4ecf506' 2024-08-20T22:10:14.3408384Z Runner group name: 'Default' 2024-08-20T22:10:14.3409256Z Machine name: 'ip-10-0-7-132' 2024-08-20T22:10:14.3426825Z Testing runner upgrade compatibility 2024-08-20T22:10:14.5455958Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:10:14.5457803Z Actions: read 2024-08-20T22:10:14.5458184Z Attestations: read 2024-08-20T22:10:14.5458680Z Checks: read 2024-08-20T22:10:14.5459092Z Contents: read 2024-08-20T22:10:14.5459545Z Deployments: read 2024-08-20T22:10:14.5459933Z Discussions: read 2024-08-20T22:10:14.5460325Z Issues: read 2024-08-20T22:10:14.5460759Z Metadata: read 2024-08-20T22:10:14.5461141Z Packages: read 2024-08-20T22:10:14.5461545Z Pages: read 2024-08-20T22:10:14.5461973Z PullRequests: read 2024-08-20T22:10:14.5462418Z RepositoryProjects: read 2024-08-20T22:10:14.5462833Z SecurityEvents: read 2024-08-20T22:10:14.5463302Z Statuses: read 2024-08-20T22:10:14.5463912Z ##[endgroup] 2024-08-20T22:10:14.5466954Z Secret source: Actions 2024-08-20T22:10:14.5467536Z Prepare workflow directory 2024-08-20T22:10:14.6390624Z Prepare all required actions 2024-08-20T22:10:14.6554718Z Getting action download info 2024-08-20T22:10:14.9283655Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:10:15.5094638Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:19.5050131Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:19.7118168Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:19.9637337Z Getting action download info 2024-08-20T22:10:20.0619056Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:20.2826090Z Getting action download info 2024-08-20T22:10:20.3717448Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:20.5305094Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:20.5307246Z ##[group] Inputs 2024-08-20T22:10:20.5307753Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:20.5317473Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:20.5327834Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:20.5329011Z sync-tag: 2024-08-20T22:10:20.5330004Z timeout-minutes: 240 2024-08-20T22:10:20.5330351Z use-gha: 2024-08-20T22:10:20.5330653Z dashboard-tag: 2024-08-20T22:10:20.5330974Z s3-bucket: gha-artifacts 2024-08-20T22:10:20.5331333Z aws-role-to-assume: 2024-08-20T22:10:20.5331673Z ##[endgroup] 2024-08-20T22:10:20.5332580Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:20.6050680Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:20.6189949Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:20.6201661Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:20.6202187Z ##[endgroup] 2024-08-20T22:10:21.7813886Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:21.7814791Z Instance Type: g5.4xlarge 2024-08-20T22:10:21.7815503Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:21.7816054Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:27.5649286Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:27.5649921Z with: 2024-08-20T22:10:27.5650633Z github-secret: *** 2024-08-20T22:10:27.5651599Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:27.5652584Z activate-with-label: false 2024-08-20T22:10:27.5652936Z label: with-ssh 2024-08-20T22:10:27.5653254Z remove-existing-keys: true 2024-08-20T22:10:27.5653615Z fail-silently: true 2024-08-20T22:10:27.5653922Z env: 2024-08-20T22:10:27.5654196Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:27.5654539Z ##[endgroup] 2024-08-20T22:10:27.6542620Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:27.6545269Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:27.8866769Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:27.9636118Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:27.9637327Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:28.0185665Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:28.0201702Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:28.0227831Z Login using: ssh ec2-user@ec2-3-89-182-17.compute-1.amazonaws.com 2024-08-20T22:10:28.0228749Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:28.0229563Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:28.0363949Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:28.0364513Z with: 2024-08-20T22:10:28.0364796Z submodules: recursive 2024-08-20T22:10:28.0365129Z fetch-depth: 0 2024-08-20T22:10:28.0365420Z env: 2024-08-20T22:10:28.0365690Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:28.0366027Z ##[endgroup] 2024-08-20T22:10:28.0556083Z ##[group]Run retry () { 2024-08-20T22:10:28.0556462Z retry () { 2024-08-20T22:10:28.0557176Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:28.0557732Z } 2024-08-20T22:10:28.0558039Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:28.0558463Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:28.0558946Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:28.0559405Z else 2024-08-20T22:10:28.0559746Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:28.0560171Z fi 2024-08-20T22:10:28.0560509Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:28.0573467Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:28.0573975Z env: 2024-08-20T22:10:28.0574248Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:28.0574580Z NO_SUDO: 2024-08-20T22:10:28.0574851Z ##[endgroup] 2024-08-20T22:10:28.0609360Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:28.1948445Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:28.1948886Z with: 2024-08-20T22:10:28.1949214Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:28.1949651Z fetch-depth: 0 2024-08-20T22:10:28.1949964Z submodules: recursive 2024-08-20T22:10:28.1950292Z quiet-checkout: true 2024-08-20T22:10:28.1950642Z repository: pytorch/pytorch 2024-08-20T22:10:28.1951123Z token: *** 2024-08-20T22:10:28.1951416Z ssh-strict: true 2024-08-20T22:10:28.1951862Z persist-credentials: true 2024-08-20T22:10:28.1952224Z clean: true 2024-08-20T22:10:28.1952559Z sparse-checkout-cone-mode: true 2024-08-20T22:10:28.1952950Z lfs: false 2024-08-20T22:10:28.1953246Z set-safe-directory: true 2024-08-20T22:10:28.1953587Z env: 2024-08-20T22:10:28.1953859Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:28.1954198Z ##[endgroup] 2024-08-20T22:10:28.2930611Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:28.2932425Z ##[group]Getting Git version info 2024-08-20T22:10:28.2933178Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:28.2934050Z [command]/usr/bin/git version 2024-08-20T22:10:28.2934413Z git version 2.40.1 2024-08-20T22:10:28.2935747Z ##[endgroup] 2024-08-20T22:10:28.2947184Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3628eaec-0ab1-4cab-806e-36aa8fbbbec5' before making global git config changes 2024-08-20T22:10:28.2948662Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:28.2949815Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:28.3002236Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:28.3005736Z ##[group]Initializing the repository 2024-08-20T22:10:28.3008573Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:28.3052345Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:28.3053269Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:28.3054103Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:28.3054679Z hint: 2024-08-20T22:10:28.3055152Z hint: git config --global init.defaultBranch 2024-08-20T22:10:28.3055631Z hint: 2024-08-20T22:10:28.3056163Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:28.3057031Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:28.3058132Z hint: 2024-08-20T22:10:28.3058524Z hint: git branch -m 2024-08-20T22:10:28.3059295Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:28.3066090Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:28.3107588Z ##[endgroup] 2024-08-20T22:10:28.3108190Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:28.3110269Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:28.3150888Z ##[endgroup] 2024-08-20T22:10:28.3151403Z ##[group]Setting up auth 2024-08-20T22:10:28.3156771Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:28.3198177Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:28.3567760Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:28.3608663Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:28.3960385Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:28.4023719Z ##[endgroup] 2024-08-20T22:10:28.4024308Z ##[group]Fetching the repository 2024-08-20T22:10:28.4029780Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:31.4682733Z remote: Enumerating objects: 1008509 2024-08-20T22:10:31.4683920Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:31.4684611Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:31.4685262Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:31.4685851Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:31.4686359Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:31.4686949Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:31.4687522Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:31.4688070Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:31.4688880Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:31.4689386Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:31.4690081Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:31.4690636Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:31.4691134Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:31.4691689Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:31.4692260Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:31.4692757Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:31.4693321Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:31.4693881Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:31.4694386Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:31.4694884Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:31.4695512Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:31.4696049Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:31.4696563Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:31.4697191Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:31.4697689Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:31.4698332Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:31.4698957Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:31.4699623Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:31.4700287Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:31.4701266Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:31.4701992Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:31.4702580Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:31.4703080Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:31.4703572Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:31.4704169Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:31.4704998Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:31.4705495Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:31.4705984Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:31.4706485Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:31.4706982Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:31.4707479Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:31.4707970Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:31.4708477Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:31.4708976Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:31.4709474Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:31.4709964Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:31.4710464Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:31.4710962Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:31.4711468Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:31.4711961Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:31.4712460Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:31.4712958Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:31.4713455Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:31.4714227Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:31.4714735Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:31.4715237Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:31.4715744Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:31.4716264Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:31.4716762Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:31.4717260Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:31.4717766Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:31.4718293Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:31.4718821Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:31.4719345Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:31.4719884Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:31.4720407Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:31.4720920Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:31.4721446Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:31.4721964Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:31.4722477Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:31.4722985Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:31.4723503Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:31.4724016Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:31.4724534Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:31.4725047Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:31.4725559Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:31.4726072Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:31.4726590Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:31.4727101Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:31.4727611Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:31.4728274Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:31.4728795Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:31.4729301Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:31.4729890Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:31.4730406Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:31.4730919Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:31.4731438Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:31.4732111Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:31.4732624Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:31.4733136Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:31.4733647Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:31.4734160Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:31.4734678Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:31.4735201Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:31.4735710Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:31.4736220Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:31.4736735Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:31.4737251Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:31.4737762Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:31.4738287Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:31.4738801Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:31.4739312Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:31.4739859Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:31.4859153Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:31.5119998Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:31.5410750Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:31.5859012Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:31.6651879Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:31.7118008Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:31.7728907Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:31.8080142Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:31.8473469Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:31.8690197Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:31.8935884Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:31.9166954Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:31.9348698Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:31.9510726Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:31.9631636Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:31.9714959Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:31.9802255Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:31.9862543Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:31.9904777Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:31.9954550Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:31.9983555Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:32.0002177Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:32.0005797Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:32.0011307Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:32.0019773Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:32.0024163Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:32.0034415Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:32.0038689Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:32.0058570Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:32.0071209Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:32.0079266Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:32.0089253Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:32.0093750Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:32.0103898Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:32.0111173Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:32.0114447Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:32.0122298Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:32.0127553Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:32.0137039Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:32.0145119Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:32.0149272Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:32.0154785Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:32.0163198Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:32.0167273Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:32.0175843Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:32.0180201Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:32.0184514Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:32.0188455Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:32.0191072Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:32.0194828Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:32.0197465Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:32.0200600Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:32.0202841Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:32.0205248Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:32.0206042Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:32.0208458Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:32.0209185Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:32.0210566Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:32.0211484Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:32.0214371Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:32.0215208Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:32.0215749Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:32.0216286Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:32.0216819Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:32.0217360Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:32.0223410Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:32.0229738Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:32.0233926Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:32.0237619Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:32.0241896Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:32.0246238Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:32.0249634Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:32.0253289Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:32.0256558Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:32.0259444Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:32.0263093Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:32.0265454Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:32.0268572Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:32.0270858Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:32.0272210Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:32.0276446Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:32.0277411Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:32.0280377Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:32.0282554Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:32.0284840Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:32.0287088Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:32.0289888Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:32.0290796Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:32.0293206Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:32.0295924Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:32.0297590Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:32.0298586Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:32.0299330Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:32.0299868Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:32.0300981Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:32.0302560Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:32.0304127Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:32.0305364Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:32.0306022Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:32.0307628Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:32.0308486Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:32.0309106Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:53.7136386Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:11:14.7466815Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:14.7506263Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:14.7512086Z ##[endgroup] 2024-08-20T22:11:14.7512908Z ##[group]Determining the checkout info 2024-08-20T22:11:14.7513682Z ##[endgroup] 2024-08-20T22:11:14.7514459Z ##[group]Checking out the ref 2024-08-20T22:11:14.7517275Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:16.4800954Z ##[endgroup] 2024-08-20T22:11:16.4801683Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:16.4805682Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:16.4867643Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:16.4907989Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:16.4948065Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:16.4984910Z ##[endgroup] 2024-08-20T22:11:16.4985431Z ##[group]Fetching submodules 2024-08-20T22:11:16.4988826Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:16.5374305Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:16.5737636Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:16.5739793Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:16.5743746Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:16.5747857Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:16.5752979Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:16.5756216Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:16.5761030Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:16.5765788Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:16.5770373Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:16.5775247Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:16.5779884Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:16.5784627Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:16.5789450Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:16.5794626Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:16.5799492Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:16.5806108Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:16.5815142Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:16.5820835Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:16.5825850Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:16.5831316Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:16.5836726Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:16.5842292Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:16.5847714Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:16.5853930Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:16.5859536Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:16.5865777Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:16.5871492Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:16.5877544Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:16.5883632Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:16.5890062Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:16.5896343Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:16.5902993Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:16.5909075Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:16.5916843Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:16.5951653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:16.9057507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:17.0847486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:17.2927468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:17.5391516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:19.5121040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:31.0717383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:31.5565979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:31.9952848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:32.6021552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:33.9129968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:35.9724027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:41.9379244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:43.3482536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:45.0651042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:46.2914230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:46.7234843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:47.0591878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:48.2307660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:48.6402451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:48.9174337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:50.6076344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:51.3512076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:52.1202426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:57.7974960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:59.7883023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:12:04.0477463Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:12:04.3278307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:13.0041814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:13.2057705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:13.4011701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:14.3362435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:14.6254630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:15.2638344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:15.6855309Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:15.7003140Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:15.7124517Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:15.7430961Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:15.7859186Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:16.9596087Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:16.9880197Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:17.0403881Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:17.1529334Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:17.1930522Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:17.8139323Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:18.1012231Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:18.1951789Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:18.1970316Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:18.1973988Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:18.1977524Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:18.1981639Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:18.1985356Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:18.2019295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:19.1785824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:19.8049295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:21.8966908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:22.9222534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:23.2578272Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:23.3709882Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:23.8916040Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:23.9636429Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:23.9790518Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:24.1343388Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:24.1819618Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:24.2293309Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:24.2628720Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:24.3182510Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:24.3347770Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:24.3371518Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:24.3401774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:37.9307381Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:37.9527894Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:38.0543276Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:38.0563042Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:38.0566197Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:38.0569853Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:38.0602440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:38.5788082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:39.7738186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:40.9596042Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:40.9618731Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:40.9622215Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:40.9625926Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:40.9629723Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:40.9633486Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:40.9637425Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:40.9641444Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:40.9645420Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:40.9679553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:41.9169717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:42.2822191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:43.5067211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:43.8352630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:44.3528535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:45.4612506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:51.7162722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:52.1248006Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:52.1480051Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:52.1917176Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:52.2086386Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:52.2106001Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:52.2140749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:52.5257818Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:52.5484179Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:52.5967971Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:52.7156461Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:52.7359294Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:52.7816147Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:52.8484509Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:52.8934085Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:52.9227288Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:53.0453393Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:53.5689101Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:53.5726303Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:53.5729660Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:53.5762732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:54.0535297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:55.0511255Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:55.0926619Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:55.1817444Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:55.1836254Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:55.1839716Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:55.1843133Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:55.1846694Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:55.1850644Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:55.1854435Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:55.1858109Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:55.1862434Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:55.1895112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:55.6455442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:56.7407074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:57.0900044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:13:03.3426976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:13:03.6443467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:13:03.8744805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:13:04.1894807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:13:11.2286036Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:13:11.2757446Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:13:11.2948428Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:13:11.4184428Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:13:11.4347976Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:13:11.4539161Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:13:11.4741710Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:13:11.4761197Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:11.4764427Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:11.4795358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:13.3993927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:14.7448859Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:14.7983219Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:15.4798470Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:15.4943891Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:15.8119161Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:15.8145109Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:15.8148071Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:15.8178903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:16.2891209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:17.3916439Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:17.4737155Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:17.4858759Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:17.5011410Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:17.5457032Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:17.5799049Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:17.6287650Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:17.6629099Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:17.6650244Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:17.6653301Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:17.6656975Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:17.6660730Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:17.6694874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:18.7434376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:18.9832688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:20.1849021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:21.1618458Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:21.1814089Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:21.2603921Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:21.2951959Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:21.2971345Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:21.3003127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:21.5407223Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:21.5452300Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:21.5817263Z Entering 'android/libs/fbjni' 2024-08-20T22:13:21.5870933Z Entering 'third_party/FP16' 2024-08-20T22:13:21.5923874Z Entering 'third_party/FXdiv' 2024-08-20T22:13:21.5975000Z Entering 'third_party/NNPACK' 2024-08-20T22:13:21.6026156Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:21.6077525Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:21.6145082Z Entering 'third_party/benchmark' 2024-08-20T22:13:21.6195604Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:21.6246411Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:21.6297699Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:21.6349039Z Entering 'third_party/cutlass' 2024-08-20T22:13:21.6407608Z Entering 'third_party/eigen' 2024-08-20T22:13:21.6460502Z Entering 'third_party/fbgemm' 2024-08-20T22:13:21.6510234Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:21.6567979Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:21.6619534Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:21.6677214Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:21.6729607Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:21.6786531Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:21.6841308Z Entering 'third_party/fmt' 2024-08-20T22:13:21.6893444Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:21.6946231Z Entering 'third_party/gloo' 2024-08-20T22:13:21.6997951Z Entering 'third_party/googletest' 2024-08-20T22:13:21.7051067Z Entering 'third_party/ideep' 2024-08-20T22:13:21.7101225Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:21.7165151Z Entering 'third_party/ittapi' 2024-08-20T22:13:21.7220116Z Entering 'third_party/kineto' 2024-08-20T22:13:21.7270867Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:21.7321355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:21.7376577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:21.7428239Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:21.7479223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:21.7528851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:21.7585294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:21.7638777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:21.7691175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:21.7743866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:21.7797243Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:21.7847433Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:21.7900745Z Entering 'third_party/mimalloc' 2024-08-20T22:13:21.7953579Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:21.8005725Z Entering 'third_party/nlohmann' 2024-08-20T22:13:21.8060001Z Entering 'third_party/onnx' 2024-08-20T22:13:21.8128242Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:21.8179519Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:21.8235994Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:21.8289517Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:21.8339977Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:21.8391112Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:21.8446649Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:21.8498190Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:21.8548811Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:21.8597853Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:21.8647140Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:21.8701572Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:21.8757997Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:21.8832540Z Entering 'third_party/pocketfft' 2024-08-20T22:13:21.8884379Z Entering 'third_party/protobuf' 2024-08-20T22:13:21.8938902Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:21.8989100Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:21.9043320Z Entering 'third_party/psimd' 2024-08-20T22:13:21.9095741Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:21.9150752Z Entering 'third_party/pybind11' 2024-08-20T22:13:21.9202676Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:21.9257021Z Entering 'third_party/sleef' 2024-08-20T22:13:21.9309031Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:21.9360705Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:21.9418194Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:21.9467316Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:21.9518427Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:21.9566768Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:21.9640486Z ##[endgroup] 2024-08-20T22:13:21.9642867Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:21.9645768Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:22.0003878Z Entering 'android/libs/fbjni' 2024-08-20T22:13:22.0077115Z Entering 'third_party/FP16' 2024-08-20T22:13:22.0144489Z Entering 'third_party/FXdiv' 2024-08-20T22:13:22.0211155Z Entering 'third_party/NNPACK' 2024-08-20T22:13:22.0279617Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:22.0345341Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:22.0428561Z Entering 'third_party/benchmark' 2024-08-20T22:13:22.0495115Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:22.0564728Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:22.0637441Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:22.0706255Z Entering 'third_party/cutlass' 2024-08-20T22:13:22.0781882Z Entering 'third_party/eigen' 2024-08-20T22:13:22.0851195Z Entering 'third_party/fbgemm' 2024-08-20T22:13:22.0921593Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:22.0988293Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:22.1056157Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:22.1128528Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:22.1194915Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:22.1264732Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:22.1334019Z Entering 'third_party/fmt' 2024-08-20T22:13:22.1399863Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:22.1466972Z Entering 'third_party/gloo' 2024-08-20T22:13:22.1533993Z Entering 'third_party/googletest' 2024-08-20T22:13:22.1599731Z Entering 'third_party/ideep' 2024-08-20T22:13:22.1664519Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:22.1742032Z Entering 'third_party/ittapi' 2024-08-20T22:13:22.1815107Z Entering 'third_party/kineto' 2024-08-20T22:13:22.1883482Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:22.1949197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:22.2017423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:22.2083382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:22.2150011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:22.2215309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:22.2285915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:22.2357249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:22.2424790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:22.2492547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:22.2562435Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:22.2629582Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:22.2697762Z Entering 'third_party/mimalloc' 2024-08-20T22:13:22.2764795Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:22.2835692Z Entering 'third_party/nlohmann' 2024-08-20T22:13:22.2903313Z Entering 'third_party/onnx' 2024-08-20T22:13:22.2983696Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:22.3051208Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:22.3124090Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:22.3190718Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:22.3257774Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:22.3324061Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:22.3396511Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:22.3464881Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:22.3531321Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:22.3603378Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:22.3668900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:22.3737838Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:22.3807886Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:22.3897561Z Entering 'third_party/pocketfft' 2024-08-20T22:13:22.3964072Z Entering 'third_party/protobuf' 2024-08-20T22:13:22.4031550Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:22.4098345Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:22.4169126Z Entering 'third_party/psimd' 2024-08-20T22:13:22.4236680Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:22.4302863Z Entering 'third_party/pybind11' 2024-08-20T22:13:22.4371273Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:22.4447634Z Entering 'third_party/sleef' 2024-08-20T22:13:22.4514345Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:22.4582187Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:22.4648467Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:22.4715277Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:22.4780876Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:22.4844985Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:22.4938759Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:22.5299202Z Entering 'android/libs/fbjni' 2024-08-20T22:13:22.5361259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:22.5383955Z Entering 'third_party/FP16' 2024-08-20T22:13:22.5448693Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:22.5470390Z Entering 'third_party/FXdiv' 2024-08-20T22:13:22.5532473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:22.5553582Z Entering 'third_party/NNPACK' 2024-08-20T22:13:22.5615948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:22.5637905Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:22.5699740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:22.5722978Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:22.5784796Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:22.5822507Z Entering 'third_party/benchmark' 2024-08-20T22:13:22.5884101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:22.5905541Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:22.5967693Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:22.5988737Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:22.6051207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:22.6073651Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:22.6136575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:22.6157758Z Entering 'third_party/cutlass' 2024-08-20T22:13:22.6219924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:22.6248703Z Entering 'third_party/eigen' 2024-08-20T22:13:22.6309811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:22.6334027Z Entering 'third_party/fbgemm' 2024-08-20T22:13:22.6396551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:22.6418785Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:22.6481093Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:22.6502073Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:22.6565596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:22.6586780Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:22.6648913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:22.6675263Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:22.6736166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:22.6756029Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:22.6816621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:22.6840258Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:22.6902714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:22.6926391Z Entering 'third_party/fmt' 2024-08-20T22:13:22.6987810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:22.7009176Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:22.7072733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:22.7093887Z Entering 'third_party/gloo' 2024-08-20T22:13:22.7156575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:22.7177984Z Entering 'third_party/googletest' 2024-08-20T22:13:22.7240423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:22.7261850Z Entering 'third_party/ideep' 2024-08-20T22:13:22.7324078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:22.7343179Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:22.7405457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:22.7435442Z Entering 'third_party/ittapi' 2024-08-20T22:13:22.7496963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:22.7518471Z Entering 'third_party/kineto' 2024-08-20T22:13:22.7582166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:22.7601914Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:22.7664480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:22.7682994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:22.7746735Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:22.7768529Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:22.7832859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:22.7853801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:22.7917550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:22.7936315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:22.7998720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:22.8017613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:22.8080346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:22.8103414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:22.8165956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:22.8188764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:22.8260593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:22.8280550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:22.8342945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:22.8364098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:22.8426101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:22.8449153Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:22.8510254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:22.8530538Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:22.8591163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:22.8614969Z Entering 'third_party/mimalloc' 2024-08-20T22:13:22.8679694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:22.8699935Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:22.8764558Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:22.8787280Z Entering 'third_party/nlohmann' 2024-08-20T22:13:22.8849770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:22.8872796Z Entering 'third_party/onnx' 2024-08-20T22:13:22.8934948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:22.8970856Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:22.9034481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:22.9054844Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:22.9116276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:22.9140718Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:22.9203523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:22.9226367Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:22.9286846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:22.9307773Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:22.9368989Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:22.9388895Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:22.9451012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:22.9469397Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:22.9530258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:22.9551565Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:22.9618429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:22.9637960Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:22.9698365Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:22.9718841Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:22.9780144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:22.9798136Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:22.9860371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:22.9882398Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:22.9944018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:22.9966983Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:23.0028176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:23.0069063Z Entering 'third_party/pocketfft' 2024-08-20T22:13:23.0131919Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:23.0153280Z Entering 'third_party/protobuf' 2024-08-20T22:13:23.0216257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:23.0239583Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:23.0300552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:23.0322612Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:23.0387935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:23.0411043Z Entering 'third_party/psimd' 2024-08-20T22:13:23.0473709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:23.0494533Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:23.0556973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:23.0578065Z Entering 'third_party/pybind11' 2024-08-20T22:13:23.0639985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:23.0661846Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:23.0723474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:23.0744676Z Entering 'third_party/sleef' 2024-08-20T22:13:23.0806459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:23.0828804Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:23.0892402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:23.0911448Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:23.0972176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:23.0994050Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:23.1055639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:23.1075312Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:23.1138027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:23.1158085Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:23.1220042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:23.1237709Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:23.1299607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:23.2212995Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:23.2577422Z Entering 'android/libs/fbjni' 2024-08-20T22:13:23.2629449Z Entering 'third_party/FP16' 2024-08-20T22:13:23.2682240Z Entering 'third_party/FXdiv' 2024-08-20T22:13:23.2735177Z Entering 'third_party/NNPACK' 2024-08-20T22:13:23.2787764Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:23.2841006Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:23.2911075Z Entering 'third_party/benchmark' 2024-08-20T22:13:23.2964878Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:23.3017812Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:23.3070787Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:23.3124042Z Entering 'third_party/cutlass' 2024-08-20T22:13:23.3184504Z Entering 'third_party/eigen' 2024-08-20T22:13:23.3239463Z Entering 'third_party/fbgemm' 2024-08-20T22:13:23.3291456Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:23.3343305Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:23.3397200Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:23.3454999Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:23.3507891Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:23.3561055Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:23.3618621Z Entering 'third_party/fmt' 2024-08-20T22:13:23.3672486Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:23.3725735Z Entering 'third_party/gloo' 2024-08-20T22:13:23.3778967Z Entering 'third_party/googletest' 2024-08-20T22:13:23.3831584Z Entering 'third_party/ideep' 2024-08-20T22:13:23.3881871Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:23.3945221Z Entering 'third_party/ittapi' 2024-08-20T22:13:23.3997223Z Entering 'third_party/kineto' 2024-08-20T22:13:23.4048638Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:23.4097944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:23.4150130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:23.4201564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:23.4253228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:23.4301515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:23.4360311Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:23.4411699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:23.4462686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:23.4514632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:23.4568055Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:23.4621189Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:23.4674466Z Entering 'third_party/mimalloc' 2024-08-20T22:13:23.4726634Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:23.4778377Z Entering 'third_party/nlohmann' 2024-08-20T22:13:23.4833644Z Entering 'third_party/onnx' 2024-08-20T22:13:23.4899717Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:23.4951289Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:23.5008343Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:23.5062864Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:23.5113024Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:23.5164001Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:23.5218055Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:23.5270519Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:23.5320141Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:23.5370213Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:23.5419491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:23.5471770Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:23.5525992Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:23.5599568Z Entering 'third_party/pocketfft' 2024-08-20T22:13:23.5655970Z Entering 'third_party/protobuf' 2024-08-20T22:13:23.5711101Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:23.5767405Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:23.5824265Z Entering 'third_party/psimd' 2024-08-20T22:13:23.5878165Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:23.5931610Z Entering 'third_party/pybind11' 2024-08-20T22:13:23.5984086Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:23.6036684Z Entering 'third_party/sleef' 2024-08-20T22:13:23.6089096Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:23.6141240Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:23.6195571Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:23.6246708Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:23.6297215Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:23.6346740Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:23.6429059Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:23.6793935Z Entering 'android/libs/fbjni' 2024-08-20T22:13:23.6845874Z Entering 'third_party/FP16' 2024-08-20T22:13:23.6896887Z Entering 'third_party/FXdiv' 2024-08-20T22:13:23.6949429Z Entering 'third_party/NNPACK' 2024-08-20T22:13:23.7001438Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:23.7058222Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:23.7125819Z Entering 'third_party/benchmark' 2024-08-20T22:13:23.7177182Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:23.7229178Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:23.7280770Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:23.7340036Z Entering 'third_party/cutlass' 2024-08-20T22:13:23.7399482Z Entering 'third_party/eigen' 2024-08-20T22:13:23.7454367Z Entering 'third_party/fbgemm' 2024-08-20T22:13:23.7507742Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:23.7558806Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:23.7611108Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:23.7667804Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:23.7717807Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:23.7771151Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:23.7824747Z Entering 'third_party/fmt' 2024-08-20T22:13:23.7876702Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:23.7929526Z Entering 'third_party/gloo' 2024-08-20T22:13:23.7980988Z Entering 'third_party/googletest' 2024-08-20T22:13:23.8037080Z Entering 'third_party/ideep' 2024-08-20T22:13:23.8086405Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:23.8145104Z Entering 'third_party/ittapi' 2024-08-20T22:13:23.8196753Z Entering 'third_party/kineto' 2024-08-20T22:13:23.8275681Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:23.8325310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:23.8377457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:23.8428755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:23.8479272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:23.8532891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:23.8587767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:23.8641924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:23.8697826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:23.8754178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:23.8812299Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:23.8863929Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:23.8917639Z Entering 'third_party/mimalloc' 2024-08-20T22:13:23.8969171Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:23.9025149Z Entering 'third_party/nlohmann' 2024-08-20T22:13:23.9078017Z Entering 'third_party/onnx' 2024-08-20T22:13:23.9142474Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:23.9194170Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:23.9252147Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:23.9307042Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:23.9357999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:23.9407815Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:23.9459142Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:23.9510000Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:23.9562458Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:23.9614551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:23.9662666Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:23.9714916Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:23.9770818Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:23.9842951Z Entering 'third_party/pocketfft' 2024-08-20T22:13:23.9895010Z Entering 'third_party/protobuf' 2024-08-20T22:13:23.9948657Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:23.9999033Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:24.0053247Z Entering 'third_party/psimd' 2024-08-20T22:13:24.0104261Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:24.0158531Z Entering 'third_party/pybind11' 2024-08-20T22:13:24.0210378Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:24.0262519Z Entering 'third_party/sleef' 2024-08-20T22:13:24.0313687Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:24.0365436Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:24.0415537Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:24.0468228Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:24.0518853Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:24.0567108Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:24.0642775Z ##[endgroup] 2024-08-20T22:13:24.0691855Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:24.0728085Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:24.0950837Z Prepare all required actions 2024-08-20T22:13:24.0951326Z Getting action download info 2024-08-20T22:13:24.2322584Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:24.2323008Z env: 2024-08-20T22:13:24.2323284Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:24.2323620Z ##[endgroup] 2024-08-20T22:13:24.2382688Z ##[group]Run set -euo pipefail 2024-08-20T22:13:24.2383119Z set -euo pipefail 2024-08-20T22:13:24.2383513Z function get_ec2_metadata() { 2024-08-20T22:13:24.2384050Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:24.2384938Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:24.2385693Z  category=$1 2024-08-20T22:13:24.2386239Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:24.2386883Z  runner_name_str=i-0dc72530ca4ecf506 2024-08-20T22:13:24.2387377Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:24.2387878Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:24.2388445Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:24.2389137Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:24.2389734Z  else 2024-08-20T22:13:24.2390230Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:24.2390804Z  fi 2024-08-20T22:13:24.2391091Z } 2024-08-20T22:13:24.2391448Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:24.2392037Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:24.2392692Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:24.2393268Z echo "system info $(uname -a)" 2024-08-20T22:13:24.2403062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:24.2403562Z env: 2024-08-20T22:13:24.2403847Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:24.2404198Z ##[endgroup] 2024-08-20T22:13:24.2533037Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:24.2608547Z instance-id: i-0dc72530ca4ecf506 2024-08-20T22:13:24.2667396Z instance-type: g5.4xlarge 2024-08-20T22:13:24.2681274Z system info Linux ip-10-0-7-132.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:24.2706639Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:24.2707576Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:24.2717377Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:24.2718027Z env: 2024-08-20T22:13:24.2718301Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:24.2718642Z ##[endgroup] 2024-08-20T22:13:24.2797501Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:24.2798105Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:24.2798644Z  echo "Docker daemon is running..."; 2024-08-20T22:13:24.2799093Z else 2024-08-20T22:13:24.2799587Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:24.2800163Z fi 2024-08-20T22:13:24.2808712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:24.2809204Z env: 2024-08-20T22:13:24.2809484Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:24.2809832Z ##[endgroup] 2024-08-20T22:13:24.2900878Z Docker daemon is running... 2024-08-20T22:13:24.2970346Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:24.2970914Z with: 2024-08-20T22:13:24.2971197Z shell: bash 2024-08-20T22:13:24.2971492Z timeout_minutes: 5 2024-08-20T22:13:24.2971815Z max_attempts: 3 2024-08-20T22:13:24.2972133Z retry_wait_seconds: 30 2024-08-20T22:13:24.2975274Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:24.2978215Z polling_interval_seconds: 1 2024-08-20T22:13:24.2978584Z warning_on_retry: true 2024-08-20T22:13:24.2978936Z continue_on_error: false 2024-08-20T22:13:24.2979272Z env: 2024-08-20T22:13:24.2979547Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:24.2979892Z AWS_RETRY_MODE: standard 2024-08-20T22:13:24.2980244Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:24.2980584Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:24.2980961Z ##[endgroup] 2024-08-20T22:13:25.4682052Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:25.4682884Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:25.4683742Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:25.4684291Z 2024-08-20T22:13:25.4684414Z Login Succeeded 2024-08-20T22:13:26.3558087Z Command completed after 1 attempt(s). 2024-08-20T22:13:26.3621864Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:26.3622582Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:26.3623284Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:26.3634492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.3634991Z env: 2024-08-20T22:13:26.3635275Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.3635623Z ##[endgroup] 2024-08-20T22:13:26.3741080Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:26.3741844Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:26.3742432Z # shellcheck disable=SC2046 2024-08-20T22:13:26.3742889Z docker stop $(docker ps -q) || true 2024-08-20T22:13:26.3743357Z # Prune all of the docker images 2024-08-20T22:13:26.3743803Z docker system prune -af 2024-08-20T22:13:26.3752806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.3753295Z env: 2024-08-20T22:13:26.3753565Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.3753906Z ##[endgroup] 2024-08-20T22:13:26.4076869Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:26.4078503Z See 'docker stop --help'. 2024-08-20T22:13:26.4079512Z 2024-08-20T22:13:26.4079956Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:26.4080712Z 2024-08-20T22:13:26.4081007Z Stop one or more running containers 2024-08-20T22:13:26.4564997Z Total reclaimed space: 0B 2024-08-20T22:13:26.4613322Z ##[group]Run set +e 2024-08-20T22:13:26.4613671Z set +e 2024-08-20T22:13:26.4614458Z set -x 2024-08-20T22:13:26.4614760Z  2024-08-20T22:13:26.4615082Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:26.4615847Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:26.4616905Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:26.4617639Z # one is returned at random 2024-08-20T22:13:26.4618173Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:26.4618670Z  2024-08-20T22:13:26.4618999Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:26.4619609Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:26.4620561Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:26.4621108Z  2024-08-20T22:13:26.4621429Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:26.4621975Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:26.4622474Z  exit 1 2024-08-20T22:13:26.4622778Z  fi 2024-08-20T22:13:26.4623059Z fi 2024-08-20T22:13:26.4623331Z  2024-08-20T22:13:26.4623687Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:26.4624193Z  # Clean up any old records first 2024-08-20T22:13:26.4624693Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:26.4625143Z fi 2024-08-20T22:13:26.4625413Z  2024-08-20T22:13:26.4625833Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:26.4626374Z cat /etc/hosts 2024-08-20T22:13:26.4635255Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.4635742Z env: 2024-08-20T22:13:26.4636025Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.4636369Z ##[endgroup] 2024-08-20T22:13:26.4666633Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:26.4672281Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:26.4673335Z ++ tail -n1 2024-08-20T22:13:26.5149631Z + RESOLVED_IP=18.160.10.36 2024-08-20T22:13:26.5150463Z + '[' -z 18.160.10.36 ']' 2024-08-20T22:13:26.5151020Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:26.5168436Z + echo '18.160.10.36 download.pytorch.org' 2024-08-20T22:13:26.5169502Z + sudo tee -a /etc/hosts 2024-08-20T22:13:26.7921357Z 18.160.10.36 download.pytorch.org 2024-08-20T22:13:26.7944144Z + cat /etc/hosts 2024-08-20T22:13:26.7954345Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:26.7966934Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:26.7967796Z 18.160.10.36 download.pytorch.org 2024-08-20T22:13:26.8127368Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:26.8127971Z with: 2024-08-20T22:13:26.8129060Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8130403Z docker-build-dir: .ci/docker 2024-08-20T22:13:26.8130784Z working-directory: . 2024-08-20T22:13:26.8131274Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8131923Z force-push: false 2024-08-20T22:13:26.8132232Z env: 2024-08-20T22:13:26.8132508Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.8132846Z ##[endgroup] 2024-08-20T22:13:26.8152288Z ##[group]Run set -ex 2024-08-20T22:13:26.8152647Z set -ex 2024-08-20T22:13:26.8152960Z  2024-08-20T22:13:26.8153725Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:26.8154690Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:26.8155485Z # job could then download the pre-built image as usual 2024-08-20T22:13:26.8156223Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:26.8156888Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8157530Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8158084Z  2024-08-20T22:13:26.8158584Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:26.8159206Z  exit 0 2024-08-20T22:13:26.8159511Z else 2024-08-20T22:13:26.8159875Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8160324Z fi 2024-08-20T22:13:26.8160620Z  2024-08-20T22:13:26.8161091Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:26.8161934Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:26.8162714Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:26.8163469Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:26.8164198Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8164894Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8165445Z else 2024-08-20T22:13:26.8165880Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:26.8166546Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8167439Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8168182Z fi 2024-08-20T22:13:26.8180801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.8181299Z env: 2024-08-20T22:13:26.8181576Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.8181924Z REPO_NAME: pytorch 2024-08-20T22:13:26.8183048Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8184240Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:26.8184739Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8185246Z ##[endgroup] 2024-08-20T22:13:26.8214829Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:26.8215346Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:26.8215733Z + echo skip=false 2024-08-20T22:13:26.8217770Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:26.8227330Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8228557Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:26.8250929Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8251747Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8253528Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8286519Z ##[group]Run set +e 2024-08-20T22:13:26.8287061Z set +e 2024-08-20T22:13:26.8287442Z set -x 2024-08-20T22:13:26.8287875Z  2024-08-20T22:13:26.8288271Z login() { 2024-08-20T22:13:26.8289178Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:26.8290154Z } 2024-08-20T22:13:26.8290557Z  2024-08-20T22:13:26.8290931Z retry () { 2024-08-20T22:13:26.8291499Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:26.8292006Z } 2024-08-20T22:13:26.8292373Z  2024-08-20T22:13:26.8292863Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:26.8293377Z  2024-08-20T22:13:26.8293944Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:26.8294819Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:26.8295414Z  exit 0 2024-08-20T22:13:26.8295774Z fi 2024-08-20T22:13:26.8296221Z  2024-08-20T22:13:26.8296816Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:26.8297767Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:26.8298621Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:26.8299389Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:26.8300097Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:26.8300686Z else 2024-08-20T22:13:26.8301216Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:26.8302018Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:26.8302665Z fi 2024-08-20T22:13:26.8303038Z  2024-08-20T22:13:26.8303433Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:26.8304073Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8304641Z  2024-08-20T22:13:26.8305438Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:26.8306266Z  exit 0 2024-08-20T22:13:26.8306685Z fi 2024-08-20T22:13:26.8307107Z  2024-08-20T22:13:26.8307640Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:26.8308688Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:26.8309629Z  exit 1 2024-08-20T22:13:26.8310027Z fi 2024-08-20T22:13:26.8310421Z  2024-08-20T22:13:26.8311023Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:26.8312081Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:26.8313044Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:26.8314452Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:26.8315673Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:26.8316547Z fi 2024-08-20T22:13:26.8317022Z  2024-08-20T22:13:26.8317471Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:26.8326018Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.8326606Z env: 2024-08-20T22:13:26.8326982Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.8327536Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:26.8328100Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:26.8329424Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8330963Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8331653Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8332473Z ##[endgroup] 2024-08-20T22:13:26.8361626Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8362456Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8363950Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:26.8365972Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:27.3725622Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:27.3726598Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:27.3727726Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:27.3728423Z 2024-08-20T22:13:27.3728605Z Login Succeeded 2024-08-20T22:13:27.3752389Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.6087659Z { 2024-08-20T22:13:27.6088675Z "schemaVersion": 2, 2024-08-20T22:13:27.6089563Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:27.6090508Z "config": { 2024-08-20T22:13:27.6091898Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:27.6092546Z "size": 48711, 2024-08-20T22:13:27.6093296Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:27.6094006Z }, 2024-08-20T22:13:27.6094796Z "layers": [ 2024-08-20T22:13:27.6095653Z { 2024-08-20T22:13:27.6096221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6097181Z "size": 28580681, 2024-08-20T22:13:27.6098092Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:27.6099106Z }, 2024-08-20T22:13:27.6099946Z { 2024-08-20T22:13:27.6100692Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6101633Z "size": 7943451, 2024-08-20T22:13:27.6102450Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:27.6103404Z }, 2024-08-20T22:13:27.6103819Z { 2024-08-20T22:13:27.6104367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6104973Z "size": 55728572, 2024-08-20T22:13:27.6105658Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:27.6106792Z }, 2024-08-20T22:13:27.6107103Z { 2024-08-20T22:13:27.6107687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6108545Z "size": 186, 2024-08-20T22:13:27.6109178Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:27.6109893Z }, 2024-08-20T22:13:27.6110255Z { 2024-08-20T22:13:27.6110774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6111482Z "size": 6886, 2024-08-20T22:13:27.6112078Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:27.6112776Z }, 2024-08-20T22:13:27.6113583Z { 2024-08-20T22:13:27.6114481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6115166Z "size": 1329001756, 2024-08-20T22:13:27.6115903Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:27.6116616Z }, 2024-08-20T22:13:27.6116943Z { 2024-08-20T22:13:27.6117512Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6118151Z "size": 62501, 2024-08-20T22:13:27.6118905Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:27.6119782Z }, 2024-08-20T22:13:27.6120122Z { 2024-08-20T22:13:27.6120722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6121457Z "size": 1684, 2024-08-20T22:13:27.6122217Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:27.6123117Z }, 2024-08-20T22:13:27.6123495Z { 2024-08-20T22:13:27.6124070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6124710Z "size": 1523, 2024-08-20T22:13:27.6125378Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:27.6126136Z }, 2024-08-20T22:13:27.6126436Z { 2024-08-20T22:13:27.6126976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6127669Z "size": 2528295403, 2024-08-20T22:13:27.6128321Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:27.6129003Z }, 2024-08-20T22:13:27.6129390Z { 2024-08-20T22:13:27.6129904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6130693Z "size": 86016, 2024-08-20T22:13:27.6131356Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:27.6132094Z }, 2024-08-20T22:13:27.6132429Z { 2024-08-20T22:13:27.6132961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6133614Z "size": 1822, 2024-08-20T22:13:27.6134247Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:27.6135109Z }, 2024-08-20T22:13:27.6135478Z { 2024-08-20T22:13:27.6135998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6136710Z "size": 246761563, 2024-08-20T22:13:27.6137388Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:27.6138057Z }, 2024-08-20T22:13:27.6138460Z { 2024-08-20T22:13:27.6138978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6139577Z "size": 544, 2024-08-20T22:13:27.6140262Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:27.6140954Z }, 2024-08-20T22:13:27.6141336Z { 2024-08-20T22:13:27.6141836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6142474Z "size": 1258, 2024-08-20T22:13:27.6143158Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:27.6143907Z }, 2024-08-20T22:13:27.6144208Z { 2024-08-20T22:13:27.6144793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6145429Z "size": 484, 2024-08-20T22:13:27.6146165Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:27.6146979Z }, 2024-08-20T22:13:27.6147318Z { 2024-08-20T22:13:27.6147834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6148503Z "size": 91711766, 2024-08-20T22:13:27.6149139Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:27.6149835Z }, 2024-08-20T22:13:27.6150278Z { 2024-08-20T22:13:27.6150773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6151446Z "size": 3390, 2024-08-20T22:13:27.6152295Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:27.6153005Z }, 2024-08-20T22:13:27.6153338Z { 2024-08-20T22:13:27.6153923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6154592Z "size": 1909, 2024-08-20T22:13:27.6155180Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:27.6155927Z }, 2024-08-20T22:13:27.6156293Z { 2024-08-20T22:13:27.6156812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6157484Z "size": 701, 2024-08-20T22:13:27.6158129Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:27.6158840Z }, 2024-08-20T22:13:27.6159232Z { 2024-08-20T22:13:27.6159735Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6160484Z "size": 2864254879, 2024-08-20T22:13:27.6161222Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:27.6161927Z }, 2024-08-20T22:13:27.6162224Z { 2024-08-20T22:13:27.6162811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6163499Z "size": 380, 2024-08-20T22:13:27.6164113Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:27.6164902Z }, 2024-08-20T22:13:27.6165242Z { 2024-08-20T22:13:27.6165836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6166443Z "size": 12900, 2024-08-20T22:13:27.6167091Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:27.6167862Z }, 2024-08-20T22:13:27.6168212Z { 2024-08-20T22:13:27.6168738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6169454Z "size": 803, 2024-08-20T22:13:27.6170153Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:27.6170894Z }, 2024-08-20T22:13:27.6171478Z { 2024-08-20T22:13:27.6171958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6172623Z "size": 106, 2024-08-20T22:13:27.6173314Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:27.6174036Z }, 2024-08-20T22:13:27.6174362Z { 2024-08-20T22:13:27.6174921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6175566Z "size": 503, 2024-08-20T22:13:27.6176201Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:27.6176897Z }, 2024-08-20T22:13:27.6177258Z { 2024-08-20T22:13:27.6177806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6178500Z "size": 121477267, 2024-08-20T22:13:27.6179122Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:27.6179805Z }, 2024-08-20T22:13:27.6180191Z { 2024-08-20T22:13:27.6180693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6181319Z "size": 109, 2024-08-20T22:13:27.6181992Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:27.6182717Z }, 2024-08-20T22:13:27.6183020Z { 2024-08-20T22:13:27.6183652Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6184322Z "size": 490, 2024-08-20T22:13:27.6184990Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:27.6185667Z }, 2024-08-20T22:13:27.6186003Z { 2024-08-20T22:13:27.6186572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6187191Z "size": 297, 2024-08-20T22:13:27.6187815Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:27.6188557Z }, 2024-08-20T22:13:27.6189023Z { 2024-08-20T22:13:27.6189507Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6190228Z "size": 103, 2024-08-20T22:13:27.6190882Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:27.6191589Z }, 2024-08-20T22:13:27.6191989Z { 2024-08-20T22:13:27.6192514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6193172Z "size": 1473, 2024-08-20T22:13:27.6193853Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:27.6194513Z }, 2024-08-20T22:13:27.6194867Z { 2024-08-20T22:13:27.6195434Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6196047Z "size": 594729516, 2024-08-20T22:13:27.6196715Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:27.6197558Z }, 2024-08-20T22:13:27.6197948Z { 2024-08-20T22:13:27.6198438Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6199133Z "size": 164, 2024-08-20T22:13:27.6199788Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:27.6200489Z }, 2024-08-20T22:13:27.6200843Z { 2024-08-20T22:13:27.6201381Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6202028Z "size": 564, 2024-08-20T22:13:27.6202666Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:27.6203377Z }, 2024-08-20T22:13:27.6203721Z { 2024-08-20T22:13:27.6204349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6204957Z "size": 35874203, 2024-08-20T22:13:27.6205613Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:27.6206393Z }, 2024-08-20T22:13:27.6206726Z { 2024-08-20T22:13:27.6207207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6207910Z "size": 104, 2024-08-20T22:13:27.6208525Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:27.6209279Z }, 2024-08-20T22:13:27.6209584Z { 2024-08-20T22:13:27.6210223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6210932Z "size": 425, 2024-08-20T22:13:27.6211562Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:27.6212276Z }, 2024-08-20T22:13:27.6212759Z { 2024-08-20T22:13:27.6213337Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6214219Z "size": 20262183, 2024-08-20T22:13:27.6214968Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:27.6215668Z }, 2024-08-20T22:13:27.6216027Z { 2024-08-20T22:13:27.6216574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6217226Z "size": 438, 2024-08-20T22:13:27.6217877Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:27.6218612Z }, 2024-08-20T22:13:27.6218956Z { 2024-08-20T22:13:27.6219497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6220189Z "size": 701, 2024-08-20T22:13:27.6220784Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:27.6221510Z }, 2024-08-20T22:13:27.6221928Z { 2024-08-20T22:13:27.6222472Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6223079Z "size": 141, 2024-08-20T22:13:27.6223772Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:27.6224511Z }, 2024-08-20T22:13:27.6224890Z { 2024-08-20T22:13:27.6225433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6226109Z "size": 135, 2024-08-20T22:13:27.6226896Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:27.6227629Z }, 2024-08-20T22:13:27.6227971Z { 2024-08-20T22:13:27.6228493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6229206Z "size": 4838068375, 2024-08-20T22:13:27.6229825Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:27.6230525Z }, 2024-08-20T22:13:27.6230935Z { 2024-08-20T22:13:27.6231493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6232120Z "size": 194, 2024-08-20T22:13:27.6232801Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:27.6233504Z }, 2024-08-20T22:13:27.6233827Z { 2024-08-20T22:13:27.6234398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6235200Z "size": 564, 2024-08-20T22:13:27.6235879Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:27.6236540Z }, 2024-08-20T22:13:27.6236901Z { 2024-08-20T22:13:27.6237468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6238118Z "size": 43162921, 2024-08-20T22:13:27.6238774Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:27.6239515Z }, 2024-08-20T22:13:27.6239859Z { 2024-08-20T22:13:27.6240406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6241060Z "size": 106, 2024-08-20T22:13:27.6241701Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:27.6242389Z }, 2024-08-20T22:13:27.6242750Z { 2024-08-20T22:13:27.6243319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6243990Z "size": 1380, 2024-08-20T22:13:27.6244795Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:27.6245500Z }, 2024-08-20T22:13:27.6245851Z { 2024-08-20T22:13:27.6246429Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6247155Z + exit 0 2024-08-20T22:13:27.6247518Z "size": 701, 2024-08-20T22:13:27.6248213Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:27.6248883Z }, 2024-08-20T22:13:27.6249225Z { 2024-08-20T22:13:27.6249816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6250574Z "size": 139, 2024-08-20T22:13:27.6251201Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:27.6251967Z }, 2024-08-20T22:13:27.6252337Z { 2024-08-20T22:13:27.6252821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6253536Z "size": 118, 2024-08-20T22:13:27.6254163Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:27.6254924Z }, 2024-08-20T22:13:27.6255226Z { 2024-08-20T22:13:27.6255749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6256457Z "size": 4324672728, 2024-08-20T22:13:27.6257115Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:27.6257782Z }, 2024-08-20T22:13:27.6258186Z { 2024-08-20T22:13:27.6258741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6259348Z "size": 174, 2024-08-20T22:13:27.6260032Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:27.6260729Z }, 2024-08-20T22:13:27.6261096Z { 2024-08-20T22:13:27.6261627Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6262272Z "size": 907, 2024-08-20T22:13:27.6262939Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:27.6263811Z }, 2024-08-20T22:13:27.6264142Z { 2024-08-20T22:13:27.6264669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6265399Z "size": 701, 2024-08-20T22:13:27.6266012Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:27.6266722Z }, 2024-08-20T22:13:27.6267137Z { 2024-08-20T22:13:27.6267686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6268292Z "size": 134, 2024-08-20T22:13:27.6268976Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:27.6269711Z }, 2024-08-20T22:13:27.6270057Z { 2024-08-20T22:13:27.6270614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6271255Z "size": 32, 2024-08-20T22:13:27.6271897Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:27.6272799Z }, 2024-08-20T22:13:27.6273119Z { 2024-08-20T22:13:27.6273667Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6274366Z "size": 158, 2024-08-20T22:13:27.6274959Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:27.6275689Z }, 2024-08-20T22:13:27.6276086Z { 2024-08-20T22:13:27.6276617Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6277246Z "size": 1841, 2024-08-20T22:13:27.6277930Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:27.6278638Z }, 2024-08-20T22:13:27.6279081Z { 2024-08-20T22:13:27.6279562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6280223Z "size": 7529776, 2024-08-20T22:13:27.6280901Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:27.6281567Z }, 2024-08-20T22:13:27.6281926Z { 2024-08-20T22:13:27.6282491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6283135Z "size": 163, 2024-08-20T22:13:27.6283738Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:27.6284470Z }, 2024-08-20T22:13:27.6284802Z { 2024-08-20T22:13:27.6285377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6286022Z "size": 7943, 2024-08-20T22:13:27.6286670Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:27.6287364Z }, 2024-08-20T22:13:27.6287747Z { 2024-08-20T22:13:27.6288242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6288878Z "size": 8068, 2024-08-20T22:13:27.6289564Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:27.6290351Z }, 2024-08-20T22:13:27.6290686Z { 2024-08-20T22:13:27.6291270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6291923Z "size": 302, 2024-08-20T22:13:27.6292526Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:27.6293273Z }, 2024-08-20T22:13:27.6293611Z { 2024-08-20T22:13:27.6294138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6294810Z "size": 7629638, 2024-08-20T22:13:27.6295446Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:27.6296140Z }, 2024-08-20T22:13:27.6296511Z { 2024-08-20T22:13:27.6297030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6297667Z "size": 108, 2024-08-20T22:13:27.6298363Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:27.6299035Z }, 2024-08-20T22:13:27.6299433Z { 2024-08-20T22:13:27.6300010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6300750Z "size": 54145661, 2024-08-20T22:13:27.6301396Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:27.6302158Z }, 2024-08-20T22:13:27.6302518Z { 2024-08-20T22:13:27.6303079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6303678Z "size": 473, 2024-08-20T22:13:27.6304328Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:27.6305070Z }, 2024-08-20T22:13:27.6305377Z { 2024-08-20T22:13:27.6305958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6306652Z "size": 1374858766, 2024-08-20T22:13:27.6307302Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:27.6307980Z }, 2024-08-20T22:13:27.6308373Z { 2024-08-20T22:13:27.6309008Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6309645Z "size": 106, 2024-08-20T22:13:27.6310302Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:27.6311042Z }, 2024-08-20T22:13:27.6311390Z { 2024-08-20T22:13:27.6311926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6312623Z "size": 559, 2024-08-20T22:13:27.6313241Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:27.6314305Z }, 2024-08-20T22:13:27.6314708Z { 2024-08-20T22:13:27.6315225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6315953Z "size": 46248488, 2024-08-20T22:13:27.6316594Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:27.6317257Z }, 2024-08-20T22:13:27.6317662Z { 2024-08-20T22:13:27.6318177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6318842Z "size": 111, 2024-08-20T22:13:27.6319478Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:27.6320203Z }, 2024-08-20T22:13:27.6320656Z { 2024-08-20T22:13:27.6321146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6321785Z "size": 32, 2024-08-20T22:13:27.6322479Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:27.6323184Z }, 2024-08-20T22:13:27.6323483Z { 2024-08-20T22:13:27.6324077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6324712Z "size": 32, 2024-08-20T22:13:27.6325356Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:27.6326072Z }, 2024-08-20T22:13:27.6326440Z { 2024-08-20T22:13:27.6326986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6327705Z "size": 32, 2024-08-20T22:13:27.6328305Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:27.6329037Z }, 2024-08-20T22:13:27.6329430Z { 2024-08-20T22:13:27.6329914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:27.6330698Z "size": 32, 2024-08-20T22:13:27.6331375Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:27.6332101Z } 2024-08-20T22:13:27.6332406Z ] 2024-08-20T22:13:27.6332823Z } 2024-08-20T22:13:27.6643746Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:27.6644273Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:27.6644809Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:27.6654257Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:27.6654754Z env: 2024-08-20T22:13:27.6655028Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:27.6656170Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.6657350Z ##[endgroup] 2024-08-20T22:13:27.6688725Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.6741839Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:27.6742420Z with: 2024-08-20T22:13:27.6743476Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.6744776Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:27.6745295Z env: 2024-08-20T22:13:27.6745574Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:27.6745920Z ##[endgroup] 2024-08-20T22:13:27.6764519Z ##[group]Run set -x 2024-08-20T22:13:27.6765025Z set -x 2024-08-20T22:13:27.6765326Z set +e 2024-08-20T22:13:27.6765632Z  2024-08-20T22:13:27.6765914Z login() { 2024-08-20T22:13:27.6766569Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:27.6767273Z } 2024-08-20T22:13:27.6767554Z  2024-08-20T22:13:27.6767863Z retry () { 2024-08-20T22:13:27.6768249Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:27.6768695Z } 2024-08-20T22:13:27.6768973Z  2024-08-20T22:13:27.6769289Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:27.6769696Z  2024-08-20T22:13:27.6769969Z set -e 2024-08-20T22:13:27.6770568Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:27.6771256Z # only pull docker image if it's not available locally 2024-08-20T22:13:27.6772018Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:27.6772717Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:27.6773156Z fi 2024-08-20T22:13:27.6781898Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:27.6782437Z env: 2024-08-20T22:13:27.6782728Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:27.6784017Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.6785494Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:27.6786042Z ##[endgroup] 2024-08-20T22:13:27.6820258Z + set +e 2024-08-20T22:13:27.6820812Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:27.6821461Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:27.6824305Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:27.6825607Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.2262559Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:28.2263570Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:28.2264430Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:28.2264971Z 2024-08-20T22:13:28.2266920Z Login Succeeded 2024-08-20T22:13:28.2286181Z + set -e 2024-08-20T22:13:28.2287548Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.2446535Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.2448694Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.4832031Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:28.4835909Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:28.4836429Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:28.4837345Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:28.4837904Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:28.4838387Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:28.4838882Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:28.4839379Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:28.4839868Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:28.4840351Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:28.4840846Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:28.4841331Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:28.4841712Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:28.4842086Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:28.4842894Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:28.4843382Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:28.4843848Z b57676e46aee: Waiting 2024-08-20T22:13:28.4844297Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:28.4844787Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:28.4845267Z a8c1e85b5e14: Waiting 2024-08-20T22:13:28.4845765Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:28.4846249Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:28.4846736Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:28.4847229Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:28.4847690Z a41a8d1c11c8: Waiting 2024-08-20T22:13:28.4848119Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:28.4848611Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:28.4849086Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:28.4849539Z d527cbbb87e3: Waiting 2024-08-20T22:13:28.4850123Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:28.4850588Z 0c1227890755: Waiting 2024-08-20T22:13:28.4851012Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:28.4851947Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:28.4852438Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:28.4852910Z 0bf61114d37b: Waiting 2024-08-20T22:13:28.4853316Z d8d1234baab3: Waiting 2024-08-20T22:13:28.4854043Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:28.4854430Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:28.4854797Z 2df849f72ba4: Waiting 2024-08-20T22:13:28.4855123Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:28.4855485Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:28.4855820Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:28.4856192Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:28.4856788Z 4373aa743caf: Waiting 2024-08-20T22:13:28.4857117Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:28.4857489Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:28.4857845Z 34c47d63c5de: Waiting 2024-08-20T22:13:28.4858178Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:28.4858547Z 019620471c25: Pulling fs layer 2024-08-20T22:13:28.4858903Z ec1e7978c1fe: Waiting 2024-08-20T22:13:28.4859222Z 1fef4bd19792: Waiting 2024-08-20T22:13:28.4859536Z 75b2abb16c68: Waiting 2024-08-20T22:13:28.4859873Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:28.4860232Z 7ed32bc8e469: Waiting 2024-08-20T22:13:28.4860562Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:28.4860922Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:28.4861260Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:28.4861653Z ca426ffc9d2b: Waiting 2024-08-20T22:13:28.4861962Z b119c4fc6c58: Waiting 2024-08-20T22:13:28.4862277Z 6efb96fc87ea: Waiting 2024-08-20T22:13:28.4862589Z 33444a5a7e01: Waiting 2024-08-20T22:13:28.4862898Z 829ed7d91e38: Waiting 2024-08-20T22:13:28.4863206Z 7cbfce999d76: Waiting 2024-08-20T22:13:28.4863563Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:28.4863947Z 385d0421d67c: Waiting 2024-08-20T22:13:28.4864270Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:28.4864765Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:28.4865142Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:28.4865515Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:28.4865884Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:28.4866237Z 5645520256fa: Waiting 2024-08-20T22:13:28.4866562Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:28.4866918Z 7754ee068b33: Waiting 2024-08-20T22:13:28.4867641Z 6658e558ebe4: Waiting 2024-08-20T22:13:28.4867968Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:28.4868487Z 0479fbac3361: Waiting 2024-08-20T22:13:28.4868795Z 33b639226eee: Waiting 2024-08-20T22:13:28.4869124Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:28.4869809Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:28.4870185Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:28.4870546Z f957c1624481: Pulling fs layer 2024-08-20T22:13:28.4870916Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:28.4871272Z f1d6874dee9c: Waiting 2024-08-20T22:13:28.4871579Z d3547bd43146: Waiting 2024-08-20T22:13:28.4871909Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:28.4872271Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:28.4872603Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:28.4873053Z 829c0e19f4b7: Waiting 2024-08-20T22:13:28.4873385Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:28.4873739Z 560cd6122a38: Waiting 2024-08-20T22:13:28.4874064Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:28.4874412Z 916c5e858016: Waiting 2024-08-20T22:13:28.4874723Z 302c8c177944: Waiting 2024-08-20T22:13:28.4875035Z 5a4eb19d7219: Waiting 2024-08-20T22:13:28.4875347Z cbb79a705353: Waiting 2024-08-20T22:13:28.4875655Z 1811ff4d3556: Waiting 2024-08-20T22:13:28.4875967Z da6a838a66c7: Waiting 2024-08-20T22:13:28.4876299Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:28.4876649Z 019620471c25: Waiting 2024-08-20T22:13:28.4876961Z c42b3602a1fb: Waiting 2024-08-20T22:13:28.4877295Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:28.4877654Z 0f085d4fa279: Waiting 2024-08-20T22:13:28.4877959Z afc429e87b59: Waiting 2024-08-20T22:13:28.4878406Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:28.4878769Z 7c47122f4e09: Waiting 2024-08-20T22:13:28.4879079Z 11fad972ef6c: Waiting 2024-08-20T22:13:28.4879418Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:28.4879784Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:28.4880101Z 78ee77eddef2: Waiting 2024-08-20T22:13:28.4880615Z 4f4fb700ef54: Waiting 2024-08-20T22:13:28.4880927Z a7a5392e2bf6: Waiting 2024-08-20T22:13:28.4881257Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:28.4881613Z b10de02255c0: Waiting 2024-08-20T22:13:28.4881934Z 768c31dca016: Waiting 2024-08-20T22:13:28.4882257Z ee1273153675: Pulling fs layer 2024-08-20T22:13:28.4882631Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:28.4882998Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:28.4883655Z ee1273153675: Waiting 2024-08-20T22:13:28.4883985Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:28.4884361Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:28.4884727Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:28.4885098Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:28.4885470Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:28.4886027Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:28.4886391Z eb09cbf3e416: Waiting 2024-08-20T22:13:28.4886707Z 5d115b29b5d5: Waiting 2024-08-20T22:13:28.4887025Z d79fd7d52698: Waiting 2024-08-20T22:13:28.4887333Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:28.4887644Z 83808b2eff4d: Waiting 2024-08-20T22:13:28.4887957Z 9752d6fbbcab: Waiting 2024-08-20T22:13:28.4888272Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:28.4888583Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:28.4888897Z e22979f9e0ee: Waiting 2024-08-20T22:13:28.6198489Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:28.6199051Z 224fe954d725: Download complete 2024-08-20T22:13:28.7006209Z d527cbbb87e3: Download complete 2024-08-20T22:13:28.7644366Z b57676e46aee: Verifying Checksum 2024-08-20T22:13:28.7644945Z b57676e46aee: Download complete 2024-08-20T22:13:28.8211900Z 7a2c55901189: Download complete 2024-08-20T22:13:28.9149160Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:28.9149754Z a41a8d1c11c8: Download complete 2024-08-20T22:13:28.9812463Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:28.9813052Z 0c1227890755: Download complete 2024-08-20T22:13:29.0488089Z d8d1234baab3: Download complete 2024-08-20T22:13:29.0894127Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:29.0894793Z 75722010b82e: Download complete 2024-08-20T22:13:29.1695719Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:29.1696164Z ec1e7978c1fe: Download complete 2024-08-20T22:13:29.2310878Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:29.9908445Z 7a2c55901189: Pull complete 2024-08-20T22:13:30.3054371Z 224fe954d725: Pull complete 2024-08-20T22:13:31.2103616Z 75722010b82e: Pull complete 2024-08-20T22:13:31.2286745Z d527cbbb87e3: Pull complete 2024-08-20T22:13:31.2483482Z b57676e46aee: Pull complete 2024-08-20T22:13:31.7567436Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:31.7568102Z ca426ffc9d2b: Download complete 2024-08-20T22:13:31.8494104Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:31.8494581Z 4373aa743caf: Download complete 2024-08-20T22:13:31.9202156Z 75b2abb16c68: Download complete 2024-08-20T22:13:31.9931480Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:32.9598708Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:32.9599340Z 0bf61114d37b: Download complete 2024-08-20T22:13:33.0341813Z 2df849f72ba4: Download complete 2024-08-20T22:13:33.1081788Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:33.1082418Z 34c47d63c5de: Download complete 2024-08-20T22:13:33.1762406Z 1fef4bd19792: Download complete 2024-08-20T22:13:42.1087221Z a8c1e85b5e14: Download complete 2024-08-20T22:13:42.1885643Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:42.1886253Z b119c4fc6c58: Download complete 2024-08-20T22:13:42.2776750Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:42.2777394Z 33b639226eee: Download complete 2024-08-20T22:13:42.3554973Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:42.3555592Z d3547bd43146: Download complete 2024-08-20T22:13:42.4544833Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:42.4545314Z 5645520256fa: Download complete 2024-08-20T22:13:42.5174760Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:13:42.5175390Z 829ed7d91e38: Download complete 2024-08-20T22:13:43.7807152Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:43.7807615Z 385d0421d67c: Download complete 2024-08-20T22:13:43.8530852Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:43.8531317Z 7cbfce999d76: Download complete 2024-08-20T22:13:43.9341264Z 6658e558ebe4: Download complete 2024-08-20T22:13:44.0125867Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:44.0126383Z 33444a5a7e01: Download complete 2024-08-20T22:13:44.0882607Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:44.0883240Z 7c47122f4e09: Download complete 2024-08-20T22:13:44.1630546Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:44.1631009Z 1811ff4d3556: Download complete 2024-08-20T22:13:50.1550723Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:50.1551304Z 7754ee068b33: Download complete 2024-08-20T22:13:50.2222423Z 0479fbac3361: Download complete 2024-08-20T22:13:50.2968166Z f1d6874dee9c: Download complete 2024-08-20T22:13:50.7170537Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:50.7171082Z afc429e87b59: Download complete 2024-08-20T22:13:50.8548127Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:50.8548655Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:51.1116054Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:51.1116528Z 829c0e19f4b7: Download complete 2024-08-20T22:13:51.1815036Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:51.1815549Z 560cd6122a38: Download complete 2024-08-20T22:13:51.2755925Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:51.2756538Z 5a4eb19d7219: Download complete 2024-08-20T22:13:51.3469732Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:51.3470208Z da6a838a66c7: Download complete 2024-08-20T22:13:54.3830782Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:54.3831322Z 7ed32bc8e469: Download complete 2024-08-20T22:13:54.4655642Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:54.5393150Z 0f085d4fa279: Download complete 2024-08-20T22:13:55.0669110Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:55.0669744Z 4a0412373305: Download complete 2024-08-20T22:13:55.1367846Z 302c8c177944: Download complete 2024-08-20T22:13:55.2036079Z 02970347dff6: Download complete 2024-08-20T22:13:55.2731044Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:55.2731630Z 916c5e858016: Download complete 2024-08-20T22:13:55.3904169Z c42b3602a1fb: Download complete 2024-08-20T22:13:55.5586425Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:55.7184913Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:55.8770478Z 0c1227890755: Pull complete 2024-08-20T22:13:56.0497180Z d8d1234baab3: Pull complete 2024-08-20T22:14:01.8701362Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:14:01.8701900Z 6efb96fc87ea: Download complete 2024-08-20T22:14:01.9882636Z f957c1624481: Download complete 2024-08-20T22:14:02.0712530Z 768c31dca016: Verifying Checksum 2024-08-20T22:14:02.0713002Z 768c31dca016: Download complete 2024-08-20T22:14:02.1608352Z 8c32d0ecebbe: Download complete 2024-08-20T22:14:02.1716999Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:14:02.1717792Z 4f4fb700ef54: Download complete 2024-08-20T22:14:02.2459037Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:14:02.2459499Z 78ee77eddef2: Download complete 2024-08-20T22:14:02.3163048Z 37513e9b43d7: Download complete 2024-08-20T22:14:02.4551968Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:14:02.4552441Z a7a5392e2bf6: Download complete 2024-08-20T22:14:02.5202786Z b10de02255c0: Verifying Checksum 2024-08-20T22:14:02.5203241Z b10de02255c0: Download complete 2024-08-20T22:14:02.6030348Z e56eeff2db93: Verifying Checksum 2024-08-20T22:14:02.6030814Z e56eeff2db93: Download complete 2024-08-20T22:14:02.6705453Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:14:02.6706056Z 70ac3e38d0fd: Download complete 2024-08-20T22:14:02.7564492Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:14:02.7564920Z eb09cbf3e416: Download complete 2024-08-20T22:14:02.8912903Z ee1273153675: Verifying Checksum 2024-08-20T22:14:02.8913391Z ee1273153675: Download complete 2024-08-20T22:14:02.9751251Z e22979f9e0ee: Download complete 2024-08-20T22:14:03.5688996Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:14:03.5689638Z 6c1f0aabce7c: Download complete 2024-08-20T22:14:03.6528125Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:29.5948217Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:29.5948825Z 5d115b29b5d5: Download complete 2024-08-20T22:14:29.6703998Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:29.6704558Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:29.7517733Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:29.7518194Z d79fd7d52698: Download complete 2024-08-20T22:14:31.0755070Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:31.0755531Z 9752d6fbbcab: Download complete 2024-08-20T22:14:31.1394990Z 83808b2eff4d: Download complete 2024-08-20T22:15:09.8258411Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:09.8261574Z cbb79a705353: Download complete 2024-08-20T22:15:10.0877559Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:15:25.5892195Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:25.8211824Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:26.0337530Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:34.6672073Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:34.8937043Z 4373aa743caf: Pull complete 2024-08-20T22:15:35.1232930Z 75b2abb16c68: Pull complete 2024-08-20T22:15:35.3523012Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:37.8907582Z 0bf61114d37b: Pull complete 2024-08-20T22:15:37.9998290Z 2df849f72ba4: Pull complete 2024-08-20T22:15:38.1112825Z 34c47d63c5de: Pull complete 2024-08-20T22:15:38.2890588Z 1fef4bd19792: Pull complete 2024-08-20T22:16:36.3833563Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:36.5102721Z b119c4fc6c58: Pull complete 2024-08-20T22:16:36.6668306Z 33b639226eee: Pull complete 2024-08-20T22:16:36.8073101Z d3547bd43146: Pull complete 2024-08-20T22:16:36.8998476Z 5645520256fa: Pull complete 2024-08-20T22:16:37.0266917Z 829ed7d91e38: Pull complete 2024-08-20T22:16:40.4577165Z 385d0421d67c: Pull complete 2024-08-20T22:16:40.6263075Z 7cbfce999d76: Pull complete 2024-08-20T22:16:40.7957747Z 6658e558ebe4: Pull complete 2024-08-20T22:16:40.9778093Z 33444a5a7e01: Pull complete 2024-08-20T22:16:41.1938273Z 7c47122f4e09: Pull complete 2024-08-20T22:16:41.3909513Z 1811ff4d3556: Pull complete 2024-08-20T22:16:52.0131616Z 7754ee068b33: Pull complete 2024-08-20T22:16:52.2329311Z 0479fbac3361: Pull complete 2024-08-20T22:16:52.4502630Z f1d6874dee9c: Pull complete 2024-08-20T22:16:53.4471135Z afc429e87b59: Pull complete 2024-08-20T22:16:53.6641432Z 019620471c25: Pull complete 2024-08-20T22:16:53.8507889Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:54.3205819Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:54.5201481Z 560cd6122a38: Pull complete 2024-08-20T22:16:54.8980895Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:55.1153302Z da6a838a66c7: Pull complete 2024-08-20T22:17:34.2235011Z cbb79a705353: Pull complete 2024-08-20T22:17:34.4444244Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:34.6594587Z 0f085d4fa279: Pull complete 2024-08-20T22:17:37.0615651Z 4a0412373305: Pull complete 2024-08-20T22:17:37.0820096Z 302c8c177944: Pull complete 2024-08-20T22:17:37.1001267Z 02970347dff6: Pull complete 2024-08-20T22:17:37.1391192Z 916c5e858016: Pull complete 2024-08-20T22:17:37.1571190Z c42b3602a1fb: Pull complete 2024-08-20T22:19:00.1024925Z 11fad972ef6c: Pull complete 2024-08-20T22:19:00.3281142Z f957c1624481: Pull complete 2024-08-20T22:19:00.5445304Z 768c31dca016: Pull complete 2024-08-20T22:19:00.9634434Z 8c32d0ecebbe: Pull complete 2024-08-20T22:19:01.1876171Z 4f4fb700ef54: Pull complete 2024-08-20T22:19:01.4165843Z 78ee77eddef2: Pull complete 2024-08-20T22:19:01.6454016Z 37513e9b43d7: Pull complete 2024-08-20T22:19:02.0075494Z a7a5392e2bf6: Pull complete 2024-08-20T22:19:02.2249166Z b10de02255c0: Pull complete 2024-08-20T22:19:02.4349525Z e56eeff2db93: Pull complete 2024-08-20T22:19:02.6119626Z 70ac3e38d0fd: Pull complete 2024-08-20T22:19:02.8328338Z eb09cbf3e416: Pull complete 2024-08-20T22:19:04.2625951Z ee1273153675: Pull complete 2024-08-20T22:19:04.4797953Z e22979f9e0ee: Pull complete 2024-08-20T22:19:06.6693193Z 6c1f0aabce7c: Pull complete 2024-08-20T22:19:06.8896857Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:20.3971413Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:20.5954263Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:20.7919941Z d79fd7d52698: Pull complete 2024-08-20T22:19:21.6649674Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:21.8904821Z 83808b2eff4d: Pull complete 2024-08-20T22:19:22.8243485Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:22.8747016Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:22.8981277Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:22.9093940Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:22.9094877Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:22.9105237Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:22.9105725Z env: 2024-08-20T22:19:22.9105999Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:22.9106352Z ##[endgroup] 2024-08-20T22:19:22.9298613Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:22.9299155Z with: 2024-08-20T22:19:22.9299436Z driver-version: 550.54.15 2024-08-20T22:19:22.9299767Z env: 2024-08-20T22:19:22.9300037Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:22.9300380Z ##[endgroup] 2024-08-20T22:19:22.9448654Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:22.9449204Z with: 2024-08-20T22:19:22.9449478Z timeout_minutes: 10 2024-08-20T22:19:22.9449796Z max_attempts: 3 2024-08-20T22:19:22.9482163Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:22.9515411Z retry_wait_seconds: 10 2024-08-20T22:19:22.9515775Z polling_interval_seconds: 1 2024-08-20T22:19:22.9516137Z warning_on_retry: true 2024-08-20T22:19:22.9516483Z continue_on_error: false 2024-08-20T22:19:22.9516809Z env: 2024-08-20T22:19:22.9517075Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:22.9517414Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:22.9517755Z ##[endgroup] 2024-08-20T22:19:23.0336532Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:23.0337296Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:23.0340113Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:23.4943215Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:23.4944055Z No packages marked for removal. 2024-08-20T22:19:23.5005752Z Dependencies resolved. 2024-08-20T22:19:23.5017553Z Nothing to do. 2024-08-20T22:19:23.5017968Z Complete! 2024-08-20T22:19:23.5919067Z + install_nvidia_driver_common 2024-08-20T22:19:23.5920832Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:23.5921395Z + lspci 2024-08-20T22:19:23.5921773Z Before installing NVIDIA driver 2024-08-20T22:19:23.6657437Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:23.6658428Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:23.6659370Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:23.6660158Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:23.6663492Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:23.6664348Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:23.6665102Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:23.6665970Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:23.6666717Z + lsmod 2024-08-20T22:19:23.6703046Z Module Size Used by 2024-08-20T22:19:23.6703650Z xt_conntrack 16384 1 2024-08-20T22:19:23.6704112Z nft_chain_nat 16384 3 2024-08-20T22:19:23.6704491Z xt_MASQUERADE 20480 1 2024-08-20T22:19:23.6704951Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:23.6705447Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:23.6706036Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:23.6706743Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:23.6707366Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:23.6707916Z xfrm_user 57344 1 2024-08-20T22:19:23.6708324Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:23.6708836Z xt_addrtype 16384 2 2024-08-20T22:19:23.6709368Z nft_compat 20480 4 2024-08-20T22:19:23.6709992Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:23.6710722Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:23.6711272Z br_netfilter 36864 0 2024-08-20T22:19:23.6711684Z bridge 307200 1 br_netfilter 2024-08-20T22:19:23.6712139Z stp 16384 1 bridge 2024-08-20T22:19:23.6712556Z llc 16384 2 bridge,stp 2024-08-20T22:19:23.6712980Z overlay 167936 0 2024-08-20T22:19:23.6713351Z tls 114688 0 2024-08-20T22:19:23.6713720Z nls_ascii 16384 1 2024-08-20T22:19:23.6714396Z nls_cp437 20480 1 2024-08-20T22:19:23.6714769Z vfat 24576 1 2024-08-20T22:19:23.6715140Z fat 86016 1 vfat 2024-08-20T22:19:23.6715745Z sunrpc 692224 1 2024-08-20T22:19:23.6716220Z ena 167936 0 2024-08-20T22:19:23.6716703Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:23.6717198Z ptp 36864 1 ena 2024-08-20T22:19:23.6717597Z pps_core 24576 1 ptp 2024-08-20T22:19:23.6718019Z aesni_intel 393216 0 2024-08-20T22:19:23.6718387Z i8042 45056 0 2024-08-20T22:19:23.6718756Z serio 28672 3 i8042 2024-08-20T22:19:23.6719174Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:23.6719697Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:23.6720193Z button 24576 0 2024-08-20T22:19:23.6720570Z sch_fq_codel 20480 17 2024-08-20T22:19:23.6720943Z dm_mod 188416 0 2024-08-20T22:19:23.6721310Z fuse 163840 1 2024-08-20T22:19:23.6721797Z configfs 57344 1 2024-08-20T22:19:23.6722295Z dax 45056 1 dm_mod 2024-08-20T22:19:23.6722820Z loop 36864 0 2024-08-20T22:19:23.6723298Z dmi_sysfs 20480 0 2024-08-20T22:19:23.6723676Z crc32_pclmul 16384 0 2024-08-20T22:19:23.6724046Z crc32c_intel 24576 0 2024-08-20T22:19:23.6724412Z efivarfs 24576 1 2024-08-20T22:19:23.6724787Z + modinfo nvidia 2024-08-20T22:19:23.6725521Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:23.6726217Z alias: char-major-195-* 2024-08-20T22:19:23.6726594Z version: 550.54.15 2024-08-20T22:19:23.6726944Z supported: external 2024-08-20T22:19:23.6727287Z license: NVIDIA 2024-08-20T22:19:23.6727714Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:23.6728188Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:23.6728654Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:23.6729212Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:23.6729705Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:23.6730204Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:23.6730863Z depends: i2c-core,drm 2024-08-20T22:19:23.6731227Z retpoline: Y 2024-08-20T22:19:23.6731534Z name: nvidia 2024-08-20T22:19:23.6732323Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:23.6733026Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:23.6733665Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:23.6734251Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:23.6734694Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:23.6735127Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:23.6735580Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:23.6736007Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:23.6736445Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:23.6736975Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:23.6737533Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:23.6738004Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:23.6738431Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:23.6738854Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:23.6739309Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:23.6739833Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:23.6740412Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:23.6740970Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:23.6741569Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:23.6742166Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:23.6742774Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:23.6743367Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:23.6743946Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:23.6744483Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:23.6745033Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:23.6745529Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:23.6745996Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:23.6746482Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:23.6746952Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:23.6747400Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:23.6747915Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:23.6748480Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:23.6748959Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:23.6749445Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:23.6749939Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:23.6750425Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:23.6750929Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:23.6751409Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:23.6751813Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:23.6752280Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:23.6752760Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:23.6753218Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:23.6753698Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:23.6754174Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:23.6754630Z parm: rm_firmware_active:charp 2024-08-20T22:19:23.6755141Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:23.6755568Z ++ command -v nvidia-smi 2024-08-20T22:19:23.6755973Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:23.6756343Z + set +e 2024-08-20T22:19:23.6756847Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:25.9338802Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:25.9339396Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:25.9340048Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:25.9340418Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:25.9340805Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:25.9341913Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:25.9342634Z + set -e 2024-08-20T22:19:25.9342942Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:25.9343519Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:25.9344208Z + post_install_nvidia_driver_common 2024-08-20T22:19:25.9345471Z + sudo modprobe nvidia 2024-08-20T22:19:26.0620475Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:26.0620917Z + lspci 2024-08-20T22:19:26.0621219Z After installing NVIDIA driver 2024-08-20T22:19:26.0726460Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:26.0727362Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:26.0728395Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:26.0729196Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:26.0730016Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:26.0730910Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:26.0731645Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:26.0732452Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:26.0733040Z + lsmod 2024-08-20T22:19:26.0761233Z Module Size Used by 2024-08-20T22:19:26.0761647Z nvidia_uvm 4706304 0 2024-08-20T22:19:26.0762199Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:26.0762775Z drm 602112 1 nvidia 2024-08-20T22:19:26.0763314Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:26.0764227Z backlight 24576 1 drm 2024-08-20T22:19:26.0764784Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:26.0765356Z xt_conntrack 16384 1 2024-08-20T22:19:26.0765861Z nft_chain_nat 16384 3 2024-08-20T22:19:26.0766254Z xt_MASQUERADE 20480 1 2024-08-20T22:19:26.0766723Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:26.0767222Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:26.0767797Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:26.0768442Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:26.0768909Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:26.0769346Z xfrm_user 57344 1 2024-08-20T22:19:26.0769738Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:26.0770159Z xt_addrtype 16384 2 2024-08-20T22:19:26.0770672Z nft_compat 20480 4 2024-08-20T22:19:26.0771132Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:26.0771746Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:26.0772291Z br_netfilter 36864 0 2024-08-20T22:19:26.0772701Z bridge 307200 1 br_netfilter 2024-08-20T22:19:26.0773150Z stp 16384 1 bridge 2024-08-20T22:19:26.0773563Z llc 16384 2 bridge,stp 2024-08-20T22:19:26.0773986Z overlay 167936 0 2024-08-20T22:19:26.0774359Z tls 114688 0 2024-08-20T22:19:26.0774730Z nls_ascii 16384 1 2024-08-20T22:19:26.0775094Z nls_cp437 20480 1 2024-08-20T22:19:26.0775467Z vfat 24576 1 2024-08-20T22:19:26.0775843Z fat 86016 1 vfat 2024-08-20T22:19:26.0776231Z sunrpc 692224 1 2024-08-20T22:19:26.0776607Z ena 167936 0 2024-08-20T22:19:26.0776980Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:26.0777382Z ptp 36864 1 ena 2024-08-20T22:19:26.0777777Z pps_core 24576 1 ptp 2024-08-20T22:19:26.0778207Z aesni_intel 393216 0 2024-08-20T22:19:26.0778593Z i8042 45056 0 2024-08-20T22:19:26.0778970Z serio 28672 3 i8042 2024-08-20T22:19:26.0779574Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:26.0780100Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:26.0780596Z button 24576 0 2024-08-20T22:19:26.0780975Z sch_fq_codel 20480 17 2024-08-20T22:19:26.0781346Z dm_mod 188416 0 2024-08-20T22:19:26.0781749Z fuse 163840 1 2024-08-20T22:19:26.0782244Z configfs 57344 1 2024-08-20T22:19:26.0782739Z dax 45056 1 dm_mod 2024-08-20T22:19:26.0783271Z loop 36864 0 2024-08-20T22:19:26.0783765Z dmi_sysfs 20480 0 2024-08-20T22:19:26.0784262Z crc32_pclmul 16384 0 2024-08-20T22:19:26.0784761Z crc32c_intel 24576 0 2024-08-20T22:19:26.0785233Z efivarfs 24576 1 2024-08-20T22:19:26.0785598Z + modinfo nvidia 2024-08-20T22:19:26.0786302Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:26.0787114Z alias: char-major-195-* 2024-08-20T22:19:26.0787626Z version: 550.54.15 2024-08-20T22:19:26.0788115Z supported: external 2024-08-20T22:19:26.0788549Z license: NVIDIA 2024-08-20T22:19:26.0788922Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:26.0789406Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:26.0789866Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:26.0790328Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:26.0790819Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:26.0791313Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:26.0791815Z depends: i2c-core,drm 2024-08-20T22:19:26.0792325Z retpoline: Y 2024-08-20T22:19:26.0792627Z name: nvidia 2024-08-20T22:19:26.0793253Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:26.0793954Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:26.0794602Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:26.0795188Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:26.0795635Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:26.0796072Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:26.0796522Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:26.0796959Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:26.0797402Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:26.0797935Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:26.0798489Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:26.0798975Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:26.0799428Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:26.0799844Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:26.0800294Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:26.0800828Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:26.0801415Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:26.0801978Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:26.0802591Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:26.0803188Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:26.0803801Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:26.0804396Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:26.0804886Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:26.0805423Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:26.0805973Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:26.0806479Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:26.0806958Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:26.0807519Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:26.0807995Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:26.0808557Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:26.0809064Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:26.0809593Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:26.0810076Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:26.0810719Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:26.0811227Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:26.0811717Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:26.0812224Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:26.0812704Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:26.0813124Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:26.0813605Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:26.0814398Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:26.0814853Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:26.0815346Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:26.0815827Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:26.0816285Z parm: rm_firmware_active:charp 2024-08-20T22:19:26.0816688Z + set +e 2024-08-20T22:19:26.0817019Z + nvidia-smi 2024-08-20T22:19:27.6524608Z Tue Aug 20 22:19:27 2024 2024-08-20T22:19:27.6525605Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:27.6526577Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:27.6527390Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:27.6528194Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:27.6530643Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:27.6531343Z | | | MIG M. | 2024-08-20T22:19:27.6531913Z |=========================================+========================+======================| 2024-08-20T22:19:27.6620111Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:27.6620951Z | 0% 29C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:27.6621588Z | | | N/A | 2024-08-20T22:19:27.6622297Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:27.6622967Z 2024-08-20T22:19:27.6623647Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:27.6624281Z | Processes: | 2024-08-20T22:19:27.6624997Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:27.6625698Z | ID ID Usage | 2024-08-20T22:19:27.6626258Z |=========================================================================================| 2024-08-20T22:19:27.6626908Z | No running processes found | 2024-08-20T22:19:27.6627775Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:28.2726351Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:29.8366209Z NVIDIA A10G 2024-08-20T22:19:30.2918527Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:30.2919862Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:30.2920287Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:30.2920719Z + set -e 2024-08-20T22:19:30.2921021Z INFO: Ignoring allowed status 0 2024-08-20T22:19:30.2927091Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:30.2931542Z + sudo yum install -y yum-utils 2024-08-20T22:19:30.6899072Z Last metadata expiration check: 0:09:35 ago on Tue Aug 20 22:09:55 2024. 2024-08-20T22:19:30.7129780Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:30.7433775Z Dependencies resolved. 2024-08-20T22:19:30.7564906Z Nothing to do. 2024-08-20T22:19:30.7565242Z Complete! 2024-08-20T22:19:30.8540757Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:30.8541846Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:30.8543140Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:31.1433152Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:31.2179721Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:31.7025863Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:31.7249962Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:31.7548156Z Dependencies resolved. 2024-08-20T22:19:31.7671419Z Nothing to do. 2024-08-20T22:19:31.7671739Z Complete! 2024-08-20T22:19:31.8859418Z + sudo systemctl restart docker 2024-08-20T22:20:26.2592618Z Tue Aug 20 22:20:26 2024 2024-08-20T22:20:26.2593483Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:26.2594324Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:26.2595146Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:26.2596436Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:26.2601423Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:26.2602181Z | | | MIG M. | 2024-08-20T22:20:26.2602749Z |=========================================+========================+======================| 2024-08-20T22:20:26.2724714Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:26.2726218Z | 0% 29C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:26.2726908Z | | | N/A | 2024-08-20T22:20:26.2727615Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:26.2728208Z 2024-08-20T22:20:26.2728876Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:26.2729666Z | Processes: | 2024-08-20T22:20:26.2730380Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:26.2731069Z | ID ID Usage | 2024-08-20T22:20:26.2731629Z |=========================================================================================| 2024-08-20T22:20:26.2732275Z | No running processes found | 2024-08-20T22:20:26.2733041Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:27.0717059Z Command completed after 1 attempt(s). 2024-08-20T22:20:27.0799230Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:27.0800008Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:27.0800703Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:27.0801356Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:27.0816350Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:27.0816850Z env: 2024-08-20T22:20:27.0817132Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:27.0817578Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:27.0818062Z ##[endgroup] 2024-08-20T22:20:27.3659139Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:27.7853493Z Collecting psutil==5.9.1 2024-08-20T22:20:27.8262581Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:27.8832325Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:27.8886871Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:27.9733866Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:28.1302316Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:28.3325157Z Prepare all required actions 2024-08-20T22:20:28.3325680Z Getting action download info 2024-08-20T22:20:28.4337498Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:28.6110709Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:28.7427885Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:28.7428367Z with: 2024-08-20T22:20:28.7428701Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:28.7429161Z s3-bucket: gha-artifacts 2024-08-20T22:20:28.7429501Z env: 2024-08-20T22:20:28.7429777Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:28.7430393Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:28.7430879Z ##[endgroup] 2024-08-20T22:20:28.7464703Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:28.7465155Z with: 2024-08-20T22:20:28.7465530Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:28.7465982Z s3-bucket: gha-artifacts 2024-08-20T22:20:28.7466337Z region: us-east-1 2024-08-20T22:20:28.7466643Z env: 2024-08-20T22:20:28.7466932Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:28.7467385Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:28.7467873Z ##[endgroup] 2024-08-20T22:20:29.2330644Z (node:52447) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:29.2331359Z 2024-08-20T22:20:29.2331620Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:29.2332346Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:29.2333281Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:29.2988501Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:29.2990774Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:36.8442658Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:36.8449040Z Artifact download has finished successfully 2024-08-20T22:20:36.8810801Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:36.8811232Z unzip -o artifacts.zip 2024-08-20T22:20:36.8821360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8821857Z env: 2024-08-20T22:20:36.8822134Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8822580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8823057Z ##[endgroup] 2024-08-20T22:20:36.9113241Z Archive: artifacts.zip 2024-08-20T22:20:36.9114571Z creating: dist/ 2024-08-20T22:20:39.0417931Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:39.0418607Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:39.0419200Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:39.0419951Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:39.0420774Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:39.0427120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:39.0428065Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:39.0429013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:39.0430010Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:39.0430983Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:39.0432104Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:39.0433815Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:39.0434844Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:39.0435842Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:39.0437393Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:39.0439119Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:39.0440983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:39.0442629Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:39.0444120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:39.0445564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:39.0446615Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:39.0447623Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:39.0487214Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:39.0527385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:39.0528835Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:39.0574426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:39.0575859Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:39.0577313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:39.0578779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:39.0580199Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:39.0581584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:39.0582980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:39.0584381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:39.0585727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:39.0587022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:39.0588264Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:39.0589491Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:39.0590727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:39.0591909Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:39.0593128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:39.0649809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:39.0710195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:39.0711567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:39.0712554Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:39.0713422Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:39.0714662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:39.0715595Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:39.0716816Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:39.0717969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:39.0719097Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:39.0720150Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:39.0721227Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:39.0722302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:39.0723385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:39.0724466Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:39.0725538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:39.0742367Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:39.0886405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:39.0887416Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:39.0888506Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:39.0889788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:39.0890947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:39.0892030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:39.0893280Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:39.0894426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:39.0895560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:39.0896694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:39.0897798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:39.0916382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:39.0999558Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:39.1000811Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:39.1001885Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:39.1003054Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:39.1003982Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:39.1004886Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:39.1005792Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:39.1007584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:39.1008576Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:39.1009585Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:39.1131107Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:39.1193370Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:39.1194253Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:39.1194959Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:39.1195793Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:39.1202036Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:39.1202974Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:39.1203898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:39.1204883Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:39.1205846Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:39.1206948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:39.1208221Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:39.1209230Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:39.1210345Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:39.1212090Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:39.1213748Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:39.1216311Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:39.1217448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:39.1218747Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:39.1220136Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:39.1221178Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:39.1222176Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:39.1262202Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:39.1301696Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:39.1303142Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:39.1348758Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:39.1350258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:39.1352128Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:39.1353603Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:39.1355017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:39.1356405Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:39.1357799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:39.1359304Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:39.1360669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:39.1361939Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:39.1363172Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:39.1364375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:39.1365592Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:39.1366774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:39.1367983Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:39.1424211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:39.1484519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:39.1485717Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:39.1486692Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:39.1487568Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:39.1488554Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:39.1489652Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:39.1490738Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:39.1491917Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:39.1493067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:39.1494139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:39.1495258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:39.1496376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:39.1497483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:39.1498590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:39.1499694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:39.1517339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:39.1581820Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:39.1583010Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:39.1584079Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:39.1585053Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:39.1585948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:39.1586842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:39.1587869Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:39.1589456Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:39.1590517Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:39.1591450Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:39.1641964Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:39.1642880Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:39.1643649Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:39.1644538Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:39.1650994Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:39.1651989Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:39.1653005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:39.1654057Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:39.1655105Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:39.1656268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:39.1657435Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:39.1658499Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:39.1659553Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:39.1661006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:39.1662431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:39.1664359Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:39.1665575Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:39.1667135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:39.1668480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:39.1669595Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:39.1670717Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:39.1710637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:39.1750415Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:39.1752102Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:39.1797497Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:39.1798961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:39.1800508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:39.1802036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:39.1803669Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:39.1805117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:39.1806569Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:39.1808020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:39.1809493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:39.1810883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:39.1812178Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:39.1813468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:39.1814930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:39.1816173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:39.1817447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:39.1872497Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:39.1933042Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:39.1934290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:39.1935341Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:39.1936281Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:39.1937247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:39.1938255Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:39.1939391Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:39.1940635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:39.1941853Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:39.1942990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:39.1944180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:39.1945523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:39.1946715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:39.1947896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:39.1949076Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:39.1950378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:39.2075951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:39.2077278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:39.2078472Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:39.2079812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:39.2081107Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:39.2082300Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:39.2083537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:39.2084775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:39.2086019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:39.2087250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:39.2088482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:39.2105471Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:39.2161099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:39.2162392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:39.2163524Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:39.2164557Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:39.2165528Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:39.2166485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:39.2167448Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:39.2169347Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:39.2170424Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:39.2171405Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:39.2275775Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:39.2319750Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:39.2320383Z creating: build/lib/ 2024-08-20T22:20:39.2408756Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:39.2864982Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:39.2874581Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:39.2882863Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:39.2891084Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:39.2891810Z inflating: build/lib/libclog.a 2024-08-20T22:20:39.2910480Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:39.2913256Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:39.2978538Z inflating: build/lib/libgtest.a 2024-08-20T22:20:39.3053765Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:39.3118667Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:39.3125846Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:39.3153630Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:39.3281054Z inflating: build/lib/libgloo.a 2024-08-20T22:20:39.3302419Z inflating: build/lib/libfmt.a 2024-08-20T22:20:39.3397671Z inflating: build/lib/libc10.so 2024-08-20T22:20:39.3399453Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:39.3401100Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:39.3421103Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:39.3926708Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:39.3944714Z inflating: build/lib/libgmock.a 2024-08-20T22:20:39.3945382Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:39.3946364Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:39.4335958Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:40.4516498Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:40.5094035Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:40.5151755Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:40.5152291Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:40.6417070Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:40.6674483Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:40.7182064Z inflating: build/lib/libkineto.a 2024-08-20T22:20:40.7366242Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:40.7409065Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:40.8103342Z inflating: build/lib/libonnx.a 2024-08-20T22:20:43.2602730Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:43.2607711Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:43.2612023Z inflating: build/lib/libshm.so 2024-08-20T22:20:45.3468057Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:45.3469940Z inflating: build/lib/libtorch.so 2024-08-20T22:20:45.3472607Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:46.1907587Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:46.3949226Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:46.4023887Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:46.4045634Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:46.4072722Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:46.4098928Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:46.4134956Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:46.4135472Z creating: build/bin/ 2024-08-20T22:20:46.4188015Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:46.4241490Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:46.4294046Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:46.4355575Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:46.4412485Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:46.4463495Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:46.4516990Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:46.4574099Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:46.4631935Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:46.4691609Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:46.4764291Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:46.4819660Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:46.4870733Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:46.4923256Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:46.4975874Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:46.5034492Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:46.5092238Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:46.5144369Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:46.5202279Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:46.5256524Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:46.5309314Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:46.5360955Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:46.5415161Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:46.5472801Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:46.5525417Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:46.5584321Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:46.5642719Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:46.5697786Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:46.5750203Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:46.5803052Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:46.5976292Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:46.6028911Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:46.6085064Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:46.6144923Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:46.6210208Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:46.6288781Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:46.6444737Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:46.6500133Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:46.6554262Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:46.6608901Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:46.6661603Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:46.6722375Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:46.6772907Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:46.6832611Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:46.7296743Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:46.7760511Z inflating: build/bin/protoc 2024-08-20T22:20:46.7816218Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:46.7871157Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:46.7926368Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:46.7980483Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:46.8035788Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:46.8091179Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:46.8143558Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:46.8200177Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:46.8543666Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:46.8900900Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:46.9273879Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:46.9330346Z inflating: build/bin/BackoffTest 2024-08-20T22:20:46.9385245Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:46.9443721Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:46.9499460Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:46.9513840Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:46.9570180Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:46.9574683Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:46.9577890Z inflating: build/bin/example_allreduce 2024-08-20T22:20:46.9635584Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:46.9706952Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:46.9709731Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:46.9780855Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:46.9789828Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:47.0149476Z inflating: build/bin/test_lazy 2024-08-20T22:20:47.1380000Z inflating: build/bin/test_api 2024-08-20T22:20:47.1457154Z inflating: build/bin/Dict_test 2024-08-20T22:20:47.1511486Z inflating: build/bin/Dimname_test 2024-08-20T22:20:47.1578733Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:47.1639613Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:47.1700730Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:47.1762687Z inflating: build/bin/atest 2024-08-20T22:20:47.1829892Z inflating: build/bin/basic 2024-08-20T22:20:47.1887356Z inflating: build/bin/broadcast_test 2024-08-20T22:20:47.1940817Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:47.2001557Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:47.2057208Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:47.2154211Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:47.2206588Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:47.2259439Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:47.2320673Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:47.2377602Z inflating: build/bin/half_test 2024-08-20T22:20:47.2477342Z inflating: build/bin/ivalue_test 2024-08-20T22:20:47.2528317Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:47.2583857Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:47.2640368Z inflating: build/bin/memory_format_test 2024-08-20T22:20:47.2694221Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:47.2748721Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:47.2806158Z inflating: build/bin/native_test 2024-08-20T22:20:47.2858696Z inflating: build/bin/operator_name_test 2024-08-20T22:20:47.2911628Z inflating: build/bin/operators_test 2024-08-20T22:20:47.2965267Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:47.3034448Z inflating: build/bin/pow_test 2024-08-20T22:20:47.3093387Z inflating: build/bin/quantized_test 2024-08-20T22:20:47.3145553Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:47.3198192Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:47.3258191Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:47.3319554Z inflating: build/bin/scalar_test 2024-08-20T22:20:47.3373514Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:47.3427706Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:47.3509131Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:47.3566067Z inflating: build/bin/test_parallel 2024-08-20T22:20:47.3569158Z inflating: build/bin/thread_init_test 2024-08-20T22:20:47.3627169Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:47.3689491Z inflating: build/bin/type_test 2024-08-20T22:20:47.3743748Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:47.3745490Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:47.3818703Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:47.3872231Z inflating: build/bin/weakref_test 2024-08-20T22:20:47.3926310Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:47.3981255Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:47.4044709Z inflating: build/bin/IListRef_test 2024-08-20T22:20:47.4156484Z inflating: build/bin/List_test 2024-08-20T22:20:47.4226602Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:47.4357173Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:47.4459352Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:47.4594907Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:47.4703890Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:47.4768632Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:47.4870112Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:47.4924256Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:47.4983066Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:47.5034988Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:47.5353241Z inflating: build/bin/op_registration_test 2024-08-20T22:20:47.5422197Z inflating: build/bin/inline_container_test 2024-08-20T22:20:47.5477713Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:47.5533840Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:47.5595817Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:47.5653805Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:47.5727287Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:47.5789775Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:47.5842242Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:47.5902953Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:47.5956595Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:47.6025409Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:47.6085343Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:47.6138293Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:47.6191936Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:47.6244175Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:47.6298937Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:47.6355193Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:47.6408055Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:47.6471742Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:47.6523671Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:47.6579408Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:47.6596098Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:47.6664386Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:47.6724229Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:47.6790959Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:47.6856266Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:47.7723354Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:47.8328703Z inflating: build/bin/test_jit 2024-08-20T22:20:47.8329166Z creating: .additional_ci_files/ 2024-08-20T22:20:47.8392260Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:47.8641820Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:47.8678562Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:47.8679073Z rm artifacts.zip 2024-08-20T22:20:47.8690604Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:47.8691120Z env: 2024-08-20T22:20:47.8691401Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.8691859Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.8692349Z ##[endgroup] 2024-08-20T22:20:47.9827117Z ##[group]Run df -H 2024-08-20T22:20:47.9827447Z df -H 2024-08-20T22:20:47.9836840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:47.9837363Z env: 2024-08-20T22:20:47.9837644Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.9838111Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.9838613Z ##[endgroup] 2024-08-20T22:20:47.9892686Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:47.9893291Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:47.9893959Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:47.9894617Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:47.9895168Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:47.9895880Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:47.9896395Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:47.9897227Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:47.9937977Z Prepare all required actions 2024-08-20T22:20:47.9938431Z Getting action download info 2024-08-20T22:20:48.1390522Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:48.1391004Z with: 2024-08-20T22:20:48.1391272Z env: 2024-08-20T22:20:48.1391563Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.1392025Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.1392530Z ##[endgroup] 2024-08-20T22:20:48.1427217Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:48.1427885Z with: 2024-08-20T22:20:48.1428162Z name: td_results 2024-08-20T22:20:48.1428493Z s3-bucket: gha-artifacts 2024-08-20T22:20:48.1428856Z region: us-east-1 2024-08-20T22:20:48.1429167Z env: 2024-08-20T22:20:48.1429449Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.1429922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.1430425Z ##[endgroup] 2024-08-20T22:20:48.6344124Z (node:52466) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:48.6345532Z 2024-08-20T22:20:48.6346071Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:48.6347537Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:48.6348881Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:48.7018424Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:48.7026022Z Artifact download has finished successfully 2024-08-20T22:20:48.7358571Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:48.7359072Z mkdir -p .additional_ci_files 2024-08-20T22:20:48.7359646Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:48.7369993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:48.7370498Z env: 2024-08-20T22:20:48.7370787Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.7371261Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.7371762Z ##[endgroup] 2024-08-20T22:20:48.7430063Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:48.7444694Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:48.7480518Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:48.7481017Z .github/scripts/parse_ref.py 2024-08-20T22:20:48.7489578Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:48.7490083Z env: 2024-08-20T22:20:48.7490363Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.7490834Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.7491318Z ##[endgroup] 2024-08-20T22:20:48.7789266Z Prepare all required actions 2024-08-20T22:20:48.7855656Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:48.7856122Z with: 2024-08-20T22:20:48.7856694Z github-token: *** 2024-08-20T22:20:48.7857005Z env: 2024-08-20T22:20:48.7857284Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.7857745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.7858235Z ##[endgroup] 2024-08-20T22:20:48.7875286Z ##[group]Run set -eux 2024-08-20T22:20:48.7875631Z set -eux 2024-08-20T22:20:48.7876225Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:48.7885333Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:48.7885839Z env: 2024-08-20T22:20:48.7886125Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:48.7886591Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:48.7887253Z GITHUB_TOKEN: *** 2024-08-20T22:20:48.7887566Z ##[endgroup] 2024-08-20T22:20:48.7917581Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0dc72530ca4ecf506 2024-08-20T22:20:50.3368919Z setting job-id=29026527621 2024-08-20T22:20:50.3369951Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:50.3585660Z Prepare all required actions 2024-08-20T22:20:50.3586116Z Getting action download info 2024-08-20T22:20:50.4902296Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:50.4902749Z with: 2024-08-20T22:20:50.4903215Z github-token: *** 2024-08-20T22:20:50.4912031Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:50.4922035Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:50.4922790Z env: 2024-08-20T22:20:50.4923063Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.4923568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.4924043Z ##[endgroup] 2024-08-20T22:20:50.4966120Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:50.4966670Z with: 2024-08-20T22:20:50.4966936Z shell: bash 2024-08-20T22:20:50.4967237Z timeout_minutes: 10 2024-08-20T22:20:50.4967554Z max_attempts: 5 2024-08-20T22:20:50.4967866Z retry_wait_seconds: 30 2024-08-20T22:20:50.4968968Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:50.4970208Z polling_interval_seconds: 1 2024-08-20T22:20:50.4970575Z warning_on_retry: true 2024-08-20T22:20:50.4970912Z continue_on_error: false 2024-08-20T22:20:50.4971246Z env: 2024-08-20T22:20:50.4971517Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.4971952Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.4972590Z GITHUB_TOKEN: *** 2024-08-20T22:20:50.4972901Z ##[endgroup] 2024-08-20T22:20:50.5739223Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:50.8131643Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:50.9597267Z Collecting requests==2.27.1 2024-08-20T22:20:50.9941914Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:51.1853037Z Collecting pyyaml==6.0.1 2024-08-20T22:20:51.1911883Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:51.2196410Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:51.5191307Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:51.5228743Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:51.5280332Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:51.5744829Z Collecting certifi>=2017.4.17 2024-08-20T22:20:51.5785693Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:51.6589036Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:51.7812063Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:52.5518888Z Command completed after 1 attempt(s). 2024-08-20T22:20:52.5581043Z ##[group]Run set -x 2024-08-20T22:20:52.5581369Z set -x 2024-08-20T22:20:52.5581662Z  2024-08-20T22:20:52.5582201Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:52.5583053Z # in runner workspace 2024-08-20T22:20:52.5583695Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:52.5593108Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:52.5593745Z env: 2024-08-20T22:20:52.5594060Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:52.5594519Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:52.5594992Z ##[endgroup] 2024-08-20T22:20:52.5625880Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:52.5868736Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:52.5869276Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:52.5869728Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:52.5870120Z  2024-08-20T22:20:52.5870654Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:52.5871319Z # in runner workspace 2024-08-20T22:20:52.5871884Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:52.5872528Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:52.5872980Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:52.5882078Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:52.5891526Z  --selected-test-configs "" \ 2024-08-20T22:20:52.5891990Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:52.5892405Z  --tag "${TAG}" \ 2024-08-20T22:20:52.5892788Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:52.5893237Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:52.5893660Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:52.5902443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:52.5902935Z env: 2024-08-20T22:20:52.5903214Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:52.5903666Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:52.5904342Z GITHUB_TOKEN: *** 2024-08-20T22:20:52.5905019Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:52.5905754Z PR_NUMBER: 2024-08-20T22:20:52.5906064Z TAG: ciflow/inductor/133712 2024-08-20T22:20:52.5906424Z EVENT_NAME: push 2024-08-20T22:20:52.5906728Z SCHEDULE: 2024-08-20T22:20:52.5907005Z HEAD_BRANCH: 2024-08-20T22:20:52.5907298Z ##[endgroup] 2024-08-20T22:20:52.5937436Z Workflow: inductor 2024-08-20T22:20:52.5938424Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:52.8679135Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:53.0227108Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:53.0227556Z echo "Filtered matrix:" 2024-08-20T22:20:53.0236630Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:53.0246243Z  2024-08-20T22:20:53.0246520Z echo 2024-08-20T22:20:53.0246895Z echo "Is the current job unstable? False" 2024-08-20T22:20:53.0247337Z  2024-08-20T22:20:53.0247605Z echo 2024-08-20T22:20:53.0247957Z echo "Is keep-going label set? False" 2024-08-20T22:20:53.0248388Z  2024-08-20T22:20:53.0248646Z echo 2024-08-20T22:20:53.0248963Z echo "Renabled issues? " 2024-08-20T22:20:53.0258084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:53.0258569Z env: 2024-08-20T22:20:53.0258843Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:53.0259288Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:53.0259760Z ##[endgroup] 2024-08-20T22:20:53.0290117Z Filtered matrix: 2024-08-20T22:20:53.0300596Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:53.0309495Z 2024-08-20T22:20:53.0309647Z Is the current job unstable? False 2024-08-20T22:20:53.0309931Z 2024-08-20T22:20:53.0310258Z Is keep-going label set? False 2024-08-20T22:20:53.0310516Z 2024-08-20T22:20:53.0310658Z Renabled issues? 2024-08-20T22:20:53.0353585Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:53.0354287Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:53.0362486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:53.0362984Z env: 2024-08-20T22:20:53.0363263Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:53.0363715Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:53.0364196Z JOB_TIMEOUT: 240 2024-08-20T22:20:53.0364495Z ##[endgroup] 2024-08-20T22:20:53.0449690Z ##[group]Run set -x 2024-08-20T22:20:53.0450173Z set -x 2024-08-20T22:20:53.0450471Z  2024-08-20T22:20:53.0450830Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:53.0451363Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:53.0451922Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:53.0452437Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:53.0453063Z else 2024-08-20T22:20:53.0453402Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:53.0453829Z fi 2024-08-20T22:20:53.0454107Z  2024-08-20T22:20:53.0454576Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:53.0455326Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:53.0456004Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:53.0456597Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:53.0457056Z container_name=$(docker run \ 2024-08-20T22:20:53.0457476Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:53.0457852Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:53.0458253Z  -e PR_NUMBER \ 2024-08-20T22:20:53.0458611Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:53.0459008Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:53.0459417Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:53.0459813Z  -e GITHUB_JOB \ 2024-08-20T22:20:53.0460169Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:53.0460556Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:53.0460963Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:53.0461361Z  -e JOB_ID \ 2024-08-20T22:20:53.0461699Z  -e JOB_NAME \ 2024-08-20T22:20:53.0462052Z  -e BASE_SHA \ 2024-08-20T22:20:53.0462398Z  -e BRANCH \ 2024-08-20T22:20:53.0462728Z  -e SHA1 \ 2024-08-20T22:20:53.0463079Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:53.0463483Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:53.0463867Z  -e SHARD_NUMBER \ 2024-08-20T22:20:53.0464277Z  -e TEST_CONFIG \ 2024-08-20T22:20:53.0464650Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:53.0465045Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:53.0465459Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:53.0465884Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:53.0466295Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:53.0466687Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:53.0467061Z  -e NO_TD \ 2024-08-20T22:20:53.0467398Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:53.0467779Z  -e PR_LABELS \ 2024-08-20T22:20:53.0468185Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:53.0468636Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:53.0469035Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:53.0469446Z  -e XLA_CUDA \ 2024-08-20T22:20:53.0469857Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:53.0470364Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:53.0470892Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:53.0471409Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:53.0471877Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:53.0472326Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:53.0472776Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:53.0473254Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:53.0473811Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:53.0474317Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:53.0474710Z  --ipc=host \ 2024-08-20T22:20:53.0475074Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:53.0475474Z  --tty \ 2024-08-20T22:20:53.0475787Z  --detach \ 2024-08-20T22:20:53.0476272Z  --name="${container_name}" \ 2024-08-20T22:20:53.0476695Z  --user jenkins \ 2024-08-20T22:20:53.0477185Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:53.0477738Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:53.0478175Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:53.0478525Z ) 2024-08-20T22:20:53.0478931Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:53.0479922Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:53.0480883Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:53.0481792Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:53.0490934Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:53.0491278Z env: 2024-08-20T22:20:53.0491556Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:53.0492014Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:53.0492612Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:53.0493102Z PR_NUMBER: 2024-08-20T22:20:53.0493427Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:53.0493845Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:53.0494201Z GITHUB_JOB: test 2024-08-20T22:20:53.0494511Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:53.0494882Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:53.0495236Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:53.0495569Z JOB_ID: 29026527621 2024-08-20T22:20:53.0496252Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:53.0496996Z BRANCH: 2024-08-20T22:20:53.0497338Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:53.0497860Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:53.0498338Z TEST_CONFIG: inductor_huggingface 2024-08-20T22:20:53.0498742Z SHARD_NUMBER: 1 2024-08-20T22:20:53.0499053Z NUM_TEST_SHARDS: 1 2024-08-20T22:20:53.0499375Z REENABLED_ISSUES: 2024-08-20T22:20:53.0499708Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:53.0500094Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:53.0500456Z TEST_SHOWLOCALS: False 2024-08-20T22:20:53.0500795Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:53.0501129Z NO_TD: False 2024-08-20T22:20:53.0501434Z TD_DISTRIBUTED: False 2024-08-20T22:20:53.0501855Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:53.0502333Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:53.0502718Z SHM_SIZE: 2g 2024-08-20T22:20:53.0503795Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:53.0504949Z XLA_CUDA: 2024-08-20T22:20:53.0505422Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:53.0506031Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:53.0506463Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:53.0506873Z DASHBOARD_TAG: 2024-08-20T22:20:53.0507386Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:53.0507759Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:53.0508132Z ##[endgroup] 2024-08-20T22:20:53.0538597Z + [[ inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:53.0539499Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:53.0540034Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:53.0547790Z +++ nproc --ignore=2 2024-08-20T22:20:53.0591935Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:21:04.3941260Z + container_name=cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T22:21:04.3942600Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:21:04.3945784Z + docker exec -i cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:21:04.5277621Z + echo DOCKER_CONTAINER_ID=cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T22:21:04.5281348Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:21:04.5284308Z + docker exec -t cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:21:04.9255677Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:21:05.2511261Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:21:05.2514996Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:21:05.2517609Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:21:05.2521057Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:21:05.2524256Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:21:05.2529159Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:21:05.2545755Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:21:05.2558791Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:21:05.2576282Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:21:05.2958366Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:21:05.8668133Z Installing collected packages: torch 2024-08-20T22:21:15.5995458Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:15.5998215Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:15.6000198Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:15.6866381Z + export TERM=vt100 2024-08-20T22:21:15.6868093Z + TERM=vt100 2024-08-20T22:21:15.6869057Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:15.6881147Z + source .ci/pytorch/common.sh 2024-08-20T22:21:15.6885397Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:15.6895194Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:15.6897243Z +++ declare -f -t trap_add 2024-08-20T22:21:15.6902797Z ++ set -ex 2024-08-20T22:21:15.6903260Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:15.6903767Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:15.6904277Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:15.6908225Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:15.6932167Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:15.6932656Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:15.6933058Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:15.6933442Z + shift 2024-08-20T22:21:15.6933724Z + for trap_add_name in "$@" 2024-08-20T22:21:15.6940251Z +++ trap -p EXIT 2024-08-20T22:21:15.6944430Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:15.6944805Z +++ extract_trap_cmd 2024-08-20T22:21:15.6945175Z +++ printf '%s\n' '' 2024-08-20T22:21:15.6945567Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:15.6948332Z + trap -- ' 2024-08-20T22:21:15.6948684Z cleanup_workspace' EXIT 2024-08-20T22:21:15.6949169Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:16.3704692Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:16.3728668Z + echo 'Environment variables:' 2024-08-20T22:21:16.3729183Z Environment variables: 2024-08-20T22:21:16.3729519Z + env 2024-08-20T22:21:16.3740224Z INSTALLED_DB=yes 2024-08-20T22:21:16.3740767Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:16.3741275Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:16.3741832Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:16.3742537Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:16.3743299Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:16.3743876Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:16.3744573Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:16.3745262Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:16.3745765Z HOSTNAME=cdf17c7bd466 2024-08-20T22:21:16.3746624Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.3747522Z GITHUB_ACTION=__self 2024-08-20T22:21:16.3747872Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:16.3751920Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:16.3756347Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:16.3756872Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:16.3757228Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:16.3757578Z TEST_CONFIG=inductor_huggingface 2024-08-20T22:21:16.3757985Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:16.3758466Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:16.3758932Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:16.3759373Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:16.3759786Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:16.3760438Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:16.3760945Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:16.3761453Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:16.3761924Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:16.3762252Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:16.3762639Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:16.3763031Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.3763638Z XLA_CUDA= 2024-08-20T22:21:16.3764157Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:16.3766288Z *** 2024-08-20T22:21:16.3766584Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:16.3766986Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:16.3767348Z GITHUB_ACTIONS=true 2024-08-20T22:21:16.3767684Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:16.3768211Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:16.3768752Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:16.3769223Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.3769743Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:16.3770358Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.3771147Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:16.3771863Z UCC_HOME=/usr 2024-08-20T22:21:16.3772197Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:16.3772595Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:16.3772955Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:16.3773421Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:16.3773913Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:16.3774387Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:16.3774768Z SHARD_NUMBER=1 2024-08-20T22:21:16.3775078Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:16.3775421Z HOME=/var/lib/jenkins 2024-08-20T22:21:16.3775792Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:16.3776232Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:16.3776709Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:16.3777193Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:16.3777561Z CUDA_VERSION=12.1.1 2024-08-20T22:21:16.3777986Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:16.3778420Z NUM_TEST_SHARDS=1 2024-08-20T22:21:16.3778724Z UCX_HOME=/usr 2024-08-20T22:21:16.3779250Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:16.3780339Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.3781607Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:16.3782861Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.3783977Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:16.3784633Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:16.3784957Z DASHBOARD_TAG= 2024-08-20T22:21:16.3785264Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:16.3785734Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:16.3786272Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:16.3787261Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.3788173Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:16.3788579Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:16.3788951Z PR_NUMBER= 2024-08-20T22:21:16.3789226Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:16.3789558Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:16.3790004Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:16.3790452Z TERM=vt100 2024-08-20T22:21:16.3790789Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:16.3791203Z INSTALLED_VISION=yes 2024-08-20T22:21:16.3791509Z BRANCH= 2024-08-20T22:21:16.3791786Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:16.3792315Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:16.3792739Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:16.3793499Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:16.3794237Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:16.3794720Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:16.3795180Z REENABLED_ISSUES= 2024-08-20T22:21:16.3795565Z SHLVL=1 2024-08-20T22:21:16.3795819Z MAX_JOBS=14 2024-08-20T22:21:16.3796137Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:16.3796491Z NVARCH=x86_64 2024-08-20T22:21:16.3796788Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:16.3797243Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.3797775Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:16.3798262Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:16.3798914Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:16.3799490Z GITHUB_JOB=test 2024-08-20T22:21:16.3799897Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:16.3800447Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:16.3800944Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:16.3801272Z TD_DISTRIBUTED=False 2024-08-20T22:21:16.3801649Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:16.3802073Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:16.3802496Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:16.3802858Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:16.3803201Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:16.3803545Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:16.3804569Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:16.3805628Z GITHUB_BASE_REF= 2024-08-20T22:21:16.3805954Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:16.3806312Z CI=true 2024-08-20T22:21:16.3806631Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:16.3807039Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:16.3807408Z JOB_ID=29026527621 2024-08-20T22:21:16.3807716Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:16.3808043Z GITHUB_HEAD_REF= 2024-08-20T22:21:16.3808342Z GITHUB_ACTION_REF= 2024-08-20T22:21:16.3808772Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:16.3809231Z TEST_SHOWLOCALS=False 2024-08-20T22:21:16.3809564Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:16.3810010Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:16.3810920Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.3811723Z NO_TD=False 2024-08-20T22:21:16.3812020Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:16.3812379Z _=/usr/bin/env 2024-08-20T22:21:16.3812849Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:16.3984367Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:16.3985568Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:16.3986791Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:16.3987757Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:16.3988359Z + BUILD_DIR=build 2024-08-20T22:21:16.3988684Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:16.3989125Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:16.3989510Z + SHARD_NUMBER=1 2024-08-20T22:21:16.3989814Z + NUM_TEST_SHARDS=1 2024-08-20T22:21:16.3990127Z + export VALGRIND=ON 2024-08-20T22:21:16.3990435Z + VALGRIND=ON 2024-08-20T22:21:16.3990919Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:16.3991569Z + [[ 0 == \1 ]] 2024-08-20T22:21:16.3991951Z + [[ False == \1 ]] 2024-08-20T22:21:16.3992508Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:16.3993036Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:16.4004197Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:16.4005112Z + [[ -n '' ]] 2024-08-20T22:21:16.4005461Z + echo 'Environment variables' 2024-08-20T22:21:16.4005842Z Environment variables 2024-08-20T22:21:16.4006158Z + env 2024-08-20T22:21:16.4013869Z INSTALLED_DB=yes 2024-08-20T22:21:16.4014803Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:16.4015486Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:16.4016321Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:16.4017123Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:16.4017864Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:16.4018509Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:16.4019209Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:16.4019971Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:16.4020596Z HOSTNAME=cdf17c7bd466 2024-08-20T22:21:16.4021719Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.4022762Z GITHUB_ACTION=__self 2024-08-20T22:21:16.4023213Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:16.4027291Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:16.4031733Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:16.4032242Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:16.4032597Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:16.4032947Z TEST_CONFIG=inductor_huggingface 2024-08-20T22:21:16.4033346Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:16.4033812Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:16.4034269Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:16.4034713Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:16.4035125Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:16.4035516Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:16.4035963Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:16.4036451Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:16.4036907Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:16.4037234Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:16.4037609Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:16.4038006Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.4038445Z XLA_CUDA= 2024-08-20T22:21:16.4038889Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:16.4039278Z *** 2024-08-20T22:21:16.4039555Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:16.4039943Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:16.4040314Z GITHUB_ACTIONS=true 2024-08-20T22:21:16.4040645Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:16.4041138Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:16.4041670Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:16.4042140Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.4042650Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:16.4043132Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.4043912Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:16.4044666Z UCC_HOME=/usr 2024-08-20T22:21:16.4045006Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:16.4045401Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:16.4045905Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:16.4046372Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:16.4046863Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:16.4047329Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:16.4047696Z SHARD_NUMBER=1 2024-08-20T22:21:16.4048003Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:16.4048349Z HOME=/var/lib/jenkins 2024-08-20T22:21:16.4048708Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:16.4049226Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:16.4049697Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:16.4050356Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:16.4050726Z CUDA_VERSION=12.1.1 2024-08-20T22:21:16.4051151Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:16.4051585Z NUM_TEST_SHARDS=1 2024-08-20T22:21:16.4051889Z UCX_HOME=/usr 2024-08-20T22:21:16.4052397Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:16.4053476Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.4054746Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:16.4055995Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.4057104Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:16.4057739Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:16.4058068Z DASHBOARD_TAG= 2024-08-20T22:21:16.4058368Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:16.4058828Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:16.4059364Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:16.4060352Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.4061307Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:16.4061718Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:16.4062088Z PR_NUMBER= 2024-08-20T22:21:16.4062370Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:16.4062686Z VALGRIND=ON 2024-08-20T22:21:16.4062977Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:16.4063417Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:16.4063869Z TERM=vt100 2024-08-20T22:21:16.4064205Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:16.4064614Z INSTALLED_VISION=yes 2024-08-20T22:21:16.4064925Z BRANCH= 2024-08-20T22:21:16.4065209Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:16.4065599Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:16.4066017Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:16.4066770Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:16.4067511Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:16.4067984Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:16.4068454Z REENABLED_ISSUES= 2024-08-20T22:21:16.4068750Z SHLVL=1 2024-08-20T22:21:16.4069010Z MAX_JOBS=14 2024-08-20T22:21:16.4069318Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:16.4069667Z NVARCH=x86_64 2024-08-20T22:21:16.4069963Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:16.4070417Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:16.4070980Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:16.4071484Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:16.4072143Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:16.4072700Z GITHUB_JOB=test 2024-08-20T22:21:16.4073104Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:16.4073651Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:16.4074156Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:16.4074475Z TD_DISTRIBUTED=False 2024-08-20T22:21:16.4074855Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:16.4075389Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:16.4075813Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:16.4076168Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:16.4076514Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:16.4076864Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:16.4077892Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:16.4079024Z GITHUB_BASE_REF= 2024-08-20T22:21:16.4079344Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:16.4079710Z CI=true 2024-08-20T22:21:16.4080019Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:16.4080431Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:16.4080801Z JOB_ID=29026527621 2024-08-20T22:21:16.4081114Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:16.4081432Z GITHUB_HEAD_REF= 2024-08-20T22:21:16.4081732Z GITHUB_ACTION_REF= 2024-08-20T22:21:16.4082168Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:16.4082695Z TEST_SHOWLOCALS=False 2024-08-20T22:21:16.4083053Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:16.4083408Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:16.4084311Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_00d355b1-326c-4ae4-a4f1-28f2b72fc37f 2024-08-20T22:21:16.4085102Z NO_TD=False 2024-08-20T22:21:16.4085405Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:16.4085766Z _=/usr/bin/env 2024-08-20T22:21:16.4086116Z + echo 'Testing pytorch' 2024-08-20T22:21:16.4086452Z Testing pytorch 2024-08-20T22:21:16.4086775Z + export LANG=C.UTF-8 2024-08-20T22:21:16.4087111Z + LANG=C.UTF-8 2024-08-20T22:21:16.4087396Z + PR_NUMBER= 2024-08-20T22:21:16.4087735Z + [[ inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:16.4088252Z + [[ inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:16.4088752Z + [[ inductor_huggingface == \s\l\o\w ]] 2024-08-20T22:21:16.4089379Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:16.4090224Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:16.4090774Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:16.4091246Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:16.4091684Z + [[ inductor_huggingface == *crossref* ]] 2024-08-20T22:21:16.4092266Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:16.4092902Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:16.4093555Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:16.4094109Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:16.4094652Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:17.6608458Z Collecting ninja==1.10.2 2024-08-20T22:21:17.6776412Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:17.7286607Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:18.3141398Z Installing collected packages: ninja 2024-08-20T22:21:18.3221063Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:18.3222488Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:18.3668909Z Successfully installed ninja-1.10.2 2024-08-20T22:21:18.4442970Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:18.4445123Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:18.4446685Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:18.4447179Z + install_tlparse 2024-08-20T22:21:18.4447567Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:18.4448488Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:18.8729710Z Collecting tlparse==0.3.25 2024-08-20T22:21:18.8894076Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:18.9418242Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:19.5629663Z Installing collected packages: tlparse 2024-08-20T22:21:19.6416836Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:19.7206906Z ++ python -m site --user-base 2024-08-20T22:21:19.7447871Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:19.7450530Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:19.7451259Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:19.7451950Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:19.7453069Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:19.7455728Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:19.7456699Z + cd test 2024-08-20T22:21:19.7457445Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:21.3843361Z + [[ inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:21.3843976Z + [[ inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:21.3848371Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:21.3849060Z + [[ inductor_huggingface == *pr_time_benchmarks* ]] 2024-08-20T22:21:21.3849887Z + [[ inductor_huggingface == *dynamo_eager* ]] 2024-08-20T22:21:21.3850535Z + [[ inductor_huggingface == *aot_eager* ]] 2024-08-20T22:21:21.3851051Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:21:21.3851540Z + [[ inductor_huggingface == *inductor* ]] 2024-08-20T22:21:21.3851989Z + [[ inductor_huggingface != *perf* ]] 2024-08-20T22:21:21.3852659Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:21.3853102Z + [[ inductor_huggingface == *dynamic* ]] 2024-08-20T22:21:21.3853536Z + [[ inductor_huggingface == *cpu* ]] 2024-08-20T22:21:21.3854021Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:21.3883289Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:21.3886195Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:21.3886834Z + cd test 2024-08-20T22:21:21.3887317Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:22.8577953Z PyTorch built with: 2024-08-20T22:21:22.8578752Z - GCC 9.4 2024-08-20T22:21:22.8579221Z - C++ Version: 201703 2024-08-20T22:21:22.8580284Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:22.8581404Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:22.8582085Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:22.8582625Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:22.8583104Z - NNPACK is enabled 2024-08-20T22:21:22.8583497Z - CPU capability usage: AVX2 2024-08-20T22:21:22.8583923Z - CUDA Runtime 12.1 2024-08-20T22:21:22.8584447Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:22.8585067Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:22.8585515Z - Magma 2.6.1 2024-08-20T22:21:22.8592628Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:22.8599117Z 2024-08-20T22:21:23.1960791Z + cd test 2024-08-20T22:21:23.1961597Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:24.5352403Z ATen/Parallel: 2024-08-20T22:21:24.5352967Z at::get_num_threads() : 8 2024-08-20T22:21:24.5353473Z at::get_num_interop_threads() : 16 2024-08-20T22:21:24.5353998Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:24.5354405Z omp_get_max_threads() : 8 2024-08-20T22:21:24.5355465Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:24.5356280Z mkl_get_max_threads() : 8 2024-08-20T22:21:24.5356916Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:24.5357558Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:24.5358002Z Environment variables: 2024-08-20T22:21:24.5358349Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:24.5358727Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:24.5359103Z ATen parallel backend: OpenMP 2024-08-20T22:21:24.5359358Z 2024-08-20T22:21:24.8215706Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:24.8216453Z + [[ inductor_huggingface == *backward* ]] 2024-08-20T22:21:24.8216921Z + [[ inductor_huggingface == *xla* ]] 2024-08-20T22:21:24.8217373Z + [[ inductor_huggingface == *executorch* ]] 2024-08-20T22:21:24.8217883Z + [[ inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:24.8218541Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:24.8219156Z + [[ inductor_huggingface == distributed ]] 2024-08-20T22:21:24.8219818Z + [[ inductor_huggingface == *inductor_distributed* ]] 2024-08-20T22:21:24.8220535Z + [[ inductor_huggingface == *inductor-halide* ]] 2024-08-20T22:21:24.8221153Z + [[ inductor_huggingface == *inductor-micro-benchmark* ]] 2024-08-20T22:21:24.8221704Z + [[ inductor_huggingface == *huggingface* ]] 2024-08-20T22:21:24.8222139Z + install_torchvision 2024-08-20T22:21:24.8222468Z + local orig_preload 2024-08-20T22:21:24.8222789Z + local commit 2024-08-20T22:21:24.8223100Z ++ get_pinned_commit vision 2024-08-20T22:21:24.8223492Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:24.8239525Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:24.8240066Z + orig_preload= 2024-08-20T22:21:24.8240476Z + '[' -n '' ']' 2024-08-20T22:21:24.8241323Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:24.8242772Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:25.1572309Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:25.1576746Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-f685wnzj 2024-08-20T22:21:25.1607382Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-f685wnzj 2024-08-20T22:21:26.7284854Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:26.7313070Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:28.0492876Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:28.3883284Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:30.8802665Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:30.8838556Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:30.8842942Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:30.8847143Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:30.8920233Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:30.8924651Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:30.8926956Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:30.8930227Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:30.8933838Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:30.8938966Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:30.8953231Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:30.9427248Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:30.9492791Z Building wheels for collected packages: torchvision 2024-08-20T22:22:47.8333581Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:22:47.8366908Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2115809 sha256=0a8de94b758af86b22d66824020b50aa6b80a6a37ec879e2183f053ee3af2ed5 2024-08-20T22:22:47.8370091Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:47.8405700Z Successfully built torchvision 2024-08-20T22:22:48.3160798Z Installing collected packages: torchvision 2024-08-20T22:22:48.7378491Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:48.8805525Z + '[' -n '' ']' 2024-08-20T22:22:48.8805826Z + id=0 2024-08-20T22:22:48.8808328Z + test_dynamo_benchmark huggingface 0 2024-08-20T22:22:48.8809753Z ++ pwd 2024-08-20T22:22:48.8810855Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:48.8811534Z + local suite=huggingface 2024-08-20T22:22:48.8812615Z + shift 2024-08-20T22:22:48.8813197Z + local shard_id=0 2024-08-20T22:22:48.8813640Z + shift 2024-08-20T22:22:48.8814246Z + [[ inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:48.8814930Z + [[ inductor_huggingface == *perf* ]] 2024-08-20T22:22:48.8815522Z + [[ inductor_huggingface == *cpu* ]] 2024-08-20T22:22:48.8817256Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:48.8818408Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-08-20T22:22:48.8819207Z ++ pwd 2024-08-20T22:22:48.8819853Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:48.8820617Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:48.8844793Z + local name=inference 2024-08-20T22:22:48.8846132Z + shift 2024-08-20T22:22:48.8846582Z + local suite=huggingface 2024-08-20T22:22:48.8847039Z + shift 2024-08-20T22:22:48.8847387Z + local shard_id=0 2024-08-20T22:22:48.8847784Z + shift 2024-08-20T22:22:48.8848172Z + partition_flags=() 2024-08-20T22:22:48.8848620Z + local partition_flags 2024-08-20T22:22:48.8849233Z + [[ -n 1 ]] 2024-08-20T22:22:48.8849647Z + [[ -n 0 ]] 2024-08-20T22:22:48.8850548Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:48.8851445Z + [[ inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:48.8852081Z + [[ inductor_huggingface == *perf* ]] 2024-08-20T22:22:48.8852606Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:48.8853071Z + [[ inductor_huggingface == *_avx2* ]] 2024-08-20T22:22:48.8853507Z + [[ inductor_huggingface == *_avx512* ]] 2024-08-20T22:22:48.8855105Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-08-20T22:22:51.9406632Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:22:51.9408252Z warnings.warn( 2024-08-20T22:22:51.9858641Z 2024-08-20T22:22:51.9859143Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-08-20T22:51:06.9424382Z pass 2024-08-20T22:51:06.9755126Z TIMING: entire_frame_compile:25.91559 code_gen:14.21447 inductor_compile:22.29634 backend_compile:21.88737 total_wall_time:35.85608 2024-08-20T22:51:06.9757019Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:40791 | ProxyTorchDispatchMode.__torch_dispatch__:19979 | FakeTensor.__torch_dispatch__:7823 2024-08-20T22:51:06.9758179Z Dynamo produced 2 graphs covering 533 ops with 5 graph breaks (4 unique) 2024-08-20T22:51:12.2949027Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:51:12.2950678Z warnings.warn( 2024-08-20T22:51:12.5029620Z 2024-08-20T22:51:15.4374664Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:15.4375147Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:51:15.4375606Z cuda train AllenaiLongformerBase 2024-08-20T22:51:23.9105592Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T22:51:23.9107039Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T22:51:23.9109220Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-20T22:51:23.9110186Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T22:51:23.9111664Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-20T22:51:23.9112643Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:51:23.9113541Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-20T22:51:23.9115393Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:51:23.9116966Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(**cloned_inputs) 2024-08-20T22:51:23.9118916Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-08-20T22:51:23.9120477Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.longformer( 2024-08-20T22:51:23.9122206Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-08-20T22:51:23.9123770Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] encoder_outputs = self.encoder( 2024-08-20T22:51:23.9125500Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward 2024-08-20T22:51:23.9127182Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] is_global_attn = is_index_global_attn.flatten().any().item() 2024-08-20T22:51:23.9128166Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:51:23.9128891Z W0820 22:51:23.909000 9794 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:52:46.3372813Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:52:46.3376938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-08-20T22:52:46.3378132Z layer_outputs = layer_module( 2024-08-20T22:52:46.3379249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-08-20T22:52:46.3380377Z self_attn_outputs = self.attention( 2024-08-20T22:52:46.3381943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-08-20T22:52:46.3383171Z self_outputs = self.self( 2024-08-20T22:52:46.3384269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-08-20T22:52:46.3385340Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-08-20T22:52:46.3387395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-08-20T22:52:46.3388962Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-08-20T22:52:46.3390322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-08-20T22:52:46.3391459Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-08-20T22:52:46.3392116Z 2024-08-20T22:52:46.9155908Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:52:46.9157674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-08-20T22:52:46.9158808Z embedding_output = self.embeddings( 2024-08-20T22:52:46.9160120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-08-20T22:52:46.9161250Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:52:46.9161720Z 2024-08-20T22:52:46.9602663Z W0820 22:52:46.959000 9794 torch/_logging/_internal.py:1058] [16/0] Profiler function will be ignored 2024-08-20T22:53:41.5853644Z pass 2024-08-20T22:53:41.6285819Z TIMING: entire_frame_compile:106.07741 inductor_compile:72.35815 backend_compile:84.43534 code_gen:41.66474 total_wall_time:130.35572 2024-08-20T22:53:41.6288862Z STATS: call_* op count: 2366 | FakeTensorMode.__torch_dispatch__:121965 | ProxyTorchDispatchMode.__torch_dispatch__:58081 | FakeTensor.__torch_dispatch__:20150 2024-08-20T22:53:41.6291302Z Dynamo produced 7 graphs covering 2366 ops with 9 graph breaks (5 unique) 2024-08-20T22:53:50.5616799Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:53:50.5618479Z warnings.warn( 2024-08-20T22:53:50.8284540Z 2024-08-20T22:53:54.8909123Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:54.8909721Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:53:54.8910265Z cuda train BartForCausalLM 2024-08-20T22:54:09.9566069Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:54:09.9567511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T22:54:09.9568474Z return super().forward(positions + self.offset) 2024-08-20T22:54:09.9568830Z 2024-08-20T22:54:15.5698601Z pass 2024-08-20T22:54:15.6997779Z TIMING: entire_frame_compile:4.88351 inductor_compile:4.20569 backend_compile:4.09843 code_gen:2.65574 total_wall_time:6.72286 2024-08-20T22:54:15.6999243Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:4032 | ProxyTorchDispatchMode.__torch_dispatch__:1679 | FakeTensor.__torch_dispatch__:746 2024-08-20T22:54:15.7000380Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique) 2024-08-20T22:54:19.8474237Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:19.8475899Z warnings.warn( 2024-08-20T22:54:20.0218714Z 2024-08-20T22:54:27.1026984Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:27.1027632Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:54:27.1028278Z cuda train BartForConditionalGeneration 2024-08-20T22:54:52.4941939Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:54:52.4943523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T22:54:52.4944542Z return super().forward(positions + self.offset) 2024-08-20T22:54:52.4945902Z 2024-08-20T22:55:07.1645608Z pass 2024-08-20T22:55:07.3678892Z TIMING: entire_frame_compile:7.07192 inductor_compile:5.89249 backend_compile:5.42421 code_gen:3.56233 total_wall_time:9.43627 2024-08-20T22:55:07.3680533Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:8764 | ProxyTorchDispatchMode.__torch_dispatch__:3917 | FakeTensor.__torch_dispatch__:1628 2024-08-20T22:55:07.3681642Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique) 2024-08-20T22:55:11.6870070Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:11.6871730Z warnings.warn( 2024-08-20T22:55:11.9273229Z 2024-08-20T22:55:14.0599778Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:14.0600396Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:55:14.0600886Z cuda train BertForMaskedLM 2024-08-20T22:55:50.0315753Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:55:50.0317038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:55:50.0318047Z pred = mod(**cloned_inputs) 2024-08-20T22:55:50.0319347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-20T22:55:50.0320280Z outputs = self.bert( 2024-08-20T22:55:50.0321209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-20T22:55:50.0324611Z embedding_output = self.embeddings( 2024-08-20T22:55:50.0325790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-20T22:55:50.0326740Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:55:50.0327087Z 2024-08-20T22:55:50.1696616Z W0820 22:55:50.168000 10824 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:56:35.2790015Z pass 2024-08-20T22:56:35.3422131Z TIMING: entire_frame_compile:65.06664 code_gen:23.91319 inductor_compile:42.13045 backend_compile:52.00354 total_wall_time:74.9868 2024-08-20T22:56:35.3423559Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:60797 | ProxyTorchDispatchMode.__torch_dispatch__:26954 | FakeTensor.__torch_dispatch__:16929 2024-08-20T22:56:35.3424690Z Dynamo produced 2 graphs covering 995 ops with 5 graph breaks (4 unique) 2024-08-20T22:56:42.7072534Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:56:42.7074209Z warnings.warn( 2024-08-20T22:56:42.9143067Z 2024-08-20T22:56:44.8091756Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:44.8092255Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:56:44.8094110Z cuda train BertForQuestionAnswering 2024-08-20T22:57:19.7690974Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:57:19.7692384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:57:19.7693442Z pred = mod(**cloned_inputs) 2024-08-20T22:57:19.7694831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-08-20T22:57:19.7696008Z outputs = self.bert( 2024-08-20T22:57:19.7697223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-20T22:57:19.7698363Z embedding_output = self.embeddings( 2024-08-20T22:57:19.7700926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-20T22:57:19.7702023Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:57:19.7702401Z 2024-08-20T22:57:19.8992607Z W0820 22:57:19.898000 11144 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:58:02.9973584Z pass 2024-08-20T22:58:03.0627246Z TIMING: entire_frame_compile:62.94163 code_gen:21.43738 inductor_compile:38.61216 backend_compile:49.07526 total_wall_time:72.31576 2024-08-20T22:58:03.0628768Z STATS: call_* op count: 993 | FakeTensorMode.__torch_dispatch__:60416 | ProxyTorchDispatchMode.__torch_dispatch__:26767 | FakeTensor.__torch_dispatch__:16739 2024-08-20T22:58:03.0629885Z Dynamo produced 2 graphs covering 993 ops with 5 graph breaks (4 unique) 2024-08-20T22:58:10.3724596Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:10.3726230Z warnings.warn( 2024-08-20T22:58:10.5864800Z 2024-08-20T22:58:39.9522705Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:39.9523217Z loading model: 0it [00:29, ?it/s] 2024-08-20T22:58:39.9526719Z cuda train BlenderbotForCausalLM 2024-08-20T22:58:39.9530289Z Traceback (most recent call last): 2024-08-20T22:58:39.9531164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:58:39.9531923Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:58:39.9532793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:58:39.9533666Z pred = mod(**cloned_inputs) 2024-08-20T22:58:39.9534939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:39.9536109Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:39.9537399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:39.9538246Z return forward_call(*args, **kwargs) 2024-08-20T22:58:39.9539327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-08-20T22:58:39.9540292Z outputs = self.model.decoder( 2024-08-20T22:58:39.9541541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:39.9542677Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:39.9543712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:39.9544650Z return forward_call(*args, **kwargs) 2024-08-20T22:58:39.9545868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-08-20T22:58:39.9546875Z layer_outputs = decoder_layer( 2024-08-20T22:58:39.9547909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:39.9548872Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:39.9549897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:39.9550793Z return forward_call(*args, **kwargs) 2024-08-20T22:58:39.9551985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-08-20T22:58:39.9553150Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-08-20T22:58:39.9554778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:39.9555755Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:39.9556781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:39.9557692Z return forward_call(*args, **kwargs) 2024-08-20T22:58:39.9558874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-08-20T22:58:39.9560398Z query_states = self.q_proj(hidden_states) * self.scaling 2024-08-20T22:58:39.9561509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:39.9562403Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:39.9563331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:39.9564167Z return forward_call(*args, **kwargs) 2024-08-20T22:58:39.9565049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-08-20T22:58:39.9565898Z return F.linear(input, self.weight, self.bias) 2024-08-20T22:58:39.9569145Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 65377 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:58:39.9571977Z 2024-08-20T22:58:39.9572273Z The above exception was the direct cause of the following exception: 2024-08-20T22:58:39.9572721Z 2024-08-20T22:58:39.9572871Z Traceback (most recent call last): 2024-08-20T22:58:39.9573518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:58:39.9574169Z ) = runner.load_model( 2024-08-20T22:58:39.9574831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 454, in load_model 2024-08-20T22:58:39.9575590Z self.validate_model(model, example_inputs) 2024-08-20T22:58:39.9576345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:58:39.9577107Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:58:39.9577576Z RuntimeError: Eager run failed 2024-08-20T22:58:39.9577835Z 2024-08-20T22:58:39.9577951Z eager_fail_to_run 2024-08-20T22:58:43.6586394Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:43.6588033Z warnings.warn( 2024-08-20T22:58:43.8358562Z 2024-08-20T22:58:45.3593303Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:45.3594275Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:58:45.3595187Z cuda train BlenderbotSmallForCausalLM 2024-08-20T22:58:45.3805412Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-08-20T22:58:54.1079248Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:58:54.1081009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-08-20T22:58:54.1082047Z return super().forward(positions) 2024-08-20T22:58:54.1082351Z 2024-08-20T22:58:58.4042736Z pass 2024-08-20T22:58:58.4308799Z TIMING: entire_frame_compile:5.84032 inductor_compile:4.68791 backend_compile:4.79748 code_gen:3.01806 total_wall_time:7.5199 2024-08-20T22:58:58.4312155Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4665 | ProxyTorchDispatchMode.__torch_dispatch__:1916 | FakeTensor.__torch_dispatch__:792 2024-08-20T22:58:58.4314582Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:59:02.5980171Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:59:02.5982768Z warnings.warn( 2024-08-20T22:59:02.7915125Z 2024-08-20T22:59:04.7851212Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:59:04.7851777Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:59:04.7852283Z cuda train BlenderbotSmallForConditionalGeneration 2024-08-20T22:59:04.8374706Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:59:16.0476811Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:59:16.0478428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-08-20T22:59:16.0479451Z return super().forward(positions) 2024-08-20T22:59:16.0479740Z 2024-08-20T22:59:25.8272682Z pass 2024-08-20T22:59:25.8625122Z TIMING: entire_frame_compile:7.56438 inductor_compile:6.1977 backend_compile:5.95676 code_gen:3.04572 total_wall_time:10.05099 2024-08-20T22:59:25.8626718Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9971 | ProxyTorchDispatchMode.__torch_dispatch__:4378 | FakeTensor.__torch_dispatch__:1692 2024-08-20T22:59:25.8627835Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique) 2024-08-20T22:59:30.1422059Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:59:30.1424503Z warnings.warn( 2024-08-20T22:59:30.3234767Z 2024-08-20T22:59:32.5015545Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:59:32.5016054Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:59:32.5016508Z cuda train CamemBert 2024-08-20T23:00:08.3310312Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:00:08.3313227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:00:08.3314758Z pred = mod(**cloned_inputs) 2024-08-20T23:00:08.3316138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-08-20T23:00:08.3317342Z outputs = self.roberta( 2024-08-20T23:00:08.3318619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-08-20T23:00:08.3319826Z embedding_output = self.embeddings( 2024-08-20T23:00:08.3321208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-08-20T23:00:08.3322263Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:00:08.3322619Z 2024-08-20T23:00:08.4690543Z W0820 23:00:08.468000 11821 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:00:54.3174950Z pass 2024-08-20T23:00:54.3842561Z TIMING: entire_frame_compile:65.75088 code_gen:24.68014 inductor_compile:42.25107 backend_compile:52.65431 total_wall_time:75.48483 2024-08-20T23:00:54.3843972Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:60920 | ProxyTorchDispatchMode.__torch_dispatch__:27006 | FakeTensor.__torch_dispatch__:16957 2024-08-20T23:00:54.3846232Z Dynamo produced 2 graphs covering 1003 ops with 5 graph breaks (4 unique) 2024-08-20T23:01:01.6981964Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:01:01.6983580Z warnings.warn( 2024-08-20T23:01:01.9388192Z 2024-08-20T23:01:04.4926003Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:01:04.4926482Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:01:04.4928926Z cuda train DebertaForMaskedLM 2024-08-20T23:01:42.8152320Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:01:42.8154523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:01:42.8155535Z pred = mod(**cloned_inputs) 2024-08-20T23:01:42.8156782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-08-20T23:01:42.8157791Z outputs = self.deberta( 2024-08-20T23:01:42.8158836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-08-20T23:01:42.8159779Z embedding_output = self.embeddings( 2024-08-20T23:01:42.8160814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-08-20T23:01:42.8162016Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:01:42.8162409Z 2024-08-20T23:01:42.9414390Z W0820 23:01:42.940000 12044 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:02:21.5688045Z pass 2024-08-20T23:02:21.6291405Z TIMING: entire_frame_compile:60.54543 code_gen:23.62613 inductor_compile:40.10283 backend_compile:48.46755 total_wall_time:70.66839 2024-08-20T23:02:21.6292841Z STATS: call_* op count: 1318 | FakeTensorMode.__torch_dispatch__:61543 | ProxyTorchDispatchMode.__torch_dispatch__:28174 | FakeTensor.__torch_dispatch__:15337 2024-08-20T23:02:21.6295339Z Dynamo produced 2 graphs covering 1318 ops with 5 graph breaks (4 unique) 2024-08-20T23:02:28.7896875Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:02:28.7898572Z warnings.warn( 2024-08-20T23:02:28.9901137Z 2024-08-20T23:02:31.1623717Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:02:31.1624227Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:02:31.1624685Z cuda train DebertaForQuestionAnswering 2024-08-20T23:03:07.9096557Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:03:07.9100122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:03:07.9101312Z pred = mod(**cloned_inputs) 2024-08-20T23:03:07.9102760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-08-20T23:03:07.9103953Z outputs = self.deberta( 2024-08-20T23:03:07.9104966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-08-20T23:03:07.9106159Z embedding_output = self.embeddings( 2024-08-20T23:03:07.9108318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-08-20T23:03:07.9109289Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:03:07.9109637Z 2024-08-20T23:03:08.0290296Z W0820 23:03:08.028000 12383 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:03:44.1645528Z pass 2024-08-20T23:03:44.2306170Z TIMING: entire_frame_compile:57.10776 code_gen:21.36537 inductor_compile:37.38719 backend_compile:45.08008 total_wall_time:67.15535 2024-08-20T23:03:44.2307584Z STATS: call_* op count: 1316 | FakeTensorMode.__torch_dispatch__:61132 | ProxyTorchDispatchMode.__torch_dispatch__:27974 | FakeTensor.__torch_dispatch__:15143 2024-08-20T23:03:44.2309114Z Dynamo produced 2 graphs covering 1316 ops with 5 graph breaks (4 unique) 2024-08-20T23:03:51.3572033Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:03:51.3573736Z warnings.warn( 2024-08-20T23:03:51.5268708Z 2024-08-20T23:04:05.3250745Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:05.3251301Z loading model: 0it [00:13, ?it/s] 2024-08-20T23:04:05.3256753Z cuda train DebertaV2ForMaskedLM 2024-08-20T23:04:05.3257281Z pass_due_to_skip 2024-08-20T23:04:05.5503741Z TIMING: 2024-08-20T23:04:05.5504068Z STATS: call_* op count: 0 2024-08-20T23:04:05.5504621Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:04:09.2100923Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:04:09.2102669Z warnings.warn( 2024-08-20T23:04:09.4348361Z 2024-08-20T23:04:20.1931186Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:20.1931648Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:04:20.1932114Z cuda train DebertaV2ForQuestionAnswering 2024-08-20T23:04:30.8359781Z ERROR:common: 2024-08-20T23:04:30.8360303Z Traceback (most recent call last): 2024-08-20T23:04:30.8361038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in check_accuracy 2024-08-20T23:04:30.8361786Z correct_result = self.run_n_iterations( 2024-08-20T23:04:30.8362728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T23:04:30.8363682Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T23:04:30.8364949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T23:04:30.8365859Z pred = mod(**cloned_inputs) 2024-08-20T23:04:30.8367018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8368045Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8369120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8369967Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8371164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-08-20T23:04:30.8372098Z outputs = self.deberta( 2024-08-20T23:04:30.8373026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8373904Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8374846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8375776Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8377199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-08-20T23:04:30.8378427Z encoder_outputs = self.encoder( 2024-08-20T23:04:30.8379921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8380816Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8381769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8382598Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8383884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-08-20T23:04:30.8385940Z output_states = layer_module( 2024-08-20T23:04:30.8386895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8387772Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8388707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8389543Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8390599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-08-20T23:04:30.8391540Z attention_output = self.attention( 2024-08-20T23:04:30.8392525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8393406Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8394345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8395171Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8396225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-08-20T23:04:30.8397135Z self_output = self.self( 2024-08-20T23:04:30.8398071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.8398948Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.8399876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.8400691Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.8401732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-08-20T23:04:30.8402940Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-08-20T23:04:30.8403985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply 2024-08-20T23:04:30.8404858Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:04:30.8406000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-08-20T23:04:30.8406964Z output = torch.softmax(output, self.dim) 2024-08-20T23:04:30.8410213Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 66864 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T23:04:30.8689976Z eager_1st_run_OOM 2024-08-20T23:04:30.8690934Z TIMING: 2024-08-20T23:04:30.8691655Z STATS: call_* op count: 0 2024-08-20T23:04:30.8692531Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:04:34.5760154Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:04:34.5763553Z warnings.warn( 2024-08-20T23:04:34.7538869Z 2024-08-20T23:04:36.2428180Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:36.2428716Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:04:36.2429162Z cuda train DistilBertForMaskedLM 2024-08-20T23:04:55.9319394Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:04:55.9320727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:04:55.9321783Z pred = mod(**cloned_inputs) 2024-08-20T23:04:55.9323108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-08-20T23:04:55.9324181Z dlbrt_output = self.distilbert( 2024-08-20T23:04:55.9325566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-20T23:04:55.9327135Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-20T23:04:55.9328767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-20T23:04:55.9329967Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-20T23:04:55.9330582Z 2024-08-20T23:04:56.0483756Z W0820 23:04:56.047000 12627 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:05:20.8691049Z pass 2024-08-20T23:05:20.9105080Z TIMING: entire_frame_compile:36.28372 code_gen:14.15906 inductor_compile:23.87783 backend_compile:29.33785 total_wall_time:41.26572 2024-08-20T23:05:20.9108660Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:32416 | ProxyTorchDispatchMode.__torch_dispatch__:14217 | FakeTensor.__torch_dispatch__:8794 2024-08-20T23:05:20.9109808Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-08-20T23:05:26.5749013Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:05:26.5750748Z warnings.warn( 2024-08-20T23:05:26.9700607Z 2024-08-20T23:05:28.2323562Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:05:28.2324033Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:05:28.2324491Z cuda train DistilBertForQuestionAnswering 2024-08-20T23:05:47.3697502Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:05:47.3698881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:05:47.3700131Z pred = mod(**cloned_inputs) 2024-08-20T23:05:47.3701566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1124, in forward 2024-08-20T23:05:47.3702559Z distilbert_output = self.distilbert( 2024-08-20T23:05:47.3703642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-20T23:05:47.3704855Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-20T23:05:47.3706125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-20T23:05:47.3707239Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-20T23:05:47.3707707Z 2024-08-20T23:05:47.4740005Z W0820 23:05:47.473000 12987 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:06:09.6351274Z pass 2024-08-20T23:06:09.6756694Z TIMING: entire_frame_compile:33.43006 code_gen:11.4851 inductor_compile:21.10976 backend_compile:26.59864 total_wall_time:38.18841 2024-08-20T23:06:09.6759352Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:32086 | ProxyTorchDispatchMode.__torch_dispatch__:14039 | FakeTensor.__torch_dispatch__:8611 2024-08-20T23:06:09.6761472Z Dynamo produced 2 graphs covering 539 ops with 5 graph breaks (4 unique) 2024-08-20T23:06:15.2632417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:06:15.2634059Z warnings.warn( 2024-08-20T23:06:15.4513576Z 2024-08-20T23:06:17.5657205Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:06:17.5657688Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:06:17.5662104Z cuda train DistillGPT2 2024-08-20T23:06:37.2859493Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:06:37.2860790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:06:37.2861763Z pred = mod(**cloned_inputs) 2024-08-20T23:06:37.2862962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-08-20T23:06:37.2863886Z transformer_outputs = self.transformer( 2024-08-20T23:06:37.2864915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-20T23:06:37.2865802Z inputs_embeds = self.wte(input_ids) 2024-08-20T23:06:37.2866100Z 2024-08-20T23:06:37.3813014Z W0820 23:06:37.380000 13191 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:06:56.6026595Z pass 2024-08-20T23:06:56.6385078Z TIMING: entire_frame_compile:29.65138 code_gen:12.87945 inductor_compile:20.54793 backend_compile:23.81127 total_wall_time:35.03576 2024-08-20T23:06:56.6386489Z STATS: call_* op count: 571 | FakeTensorMode.__torch_dispatch__:26744 | ProxyTorchDispatchMode.__torch_dispatch__:11483 | FakeTensor.__torch_dispatch__:7064 2024-08-20T23:06:56.6387644Z Dynamo produced 2 graphs covering 571 ops with 5 graph breaks (4 unique) 2024-08-20T23:07:01.8789954Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:07:01.8791602Z warnings.warn( 2024-08-20T23:07:02.0743433Z 2024-08-20T23:07:02.0752387Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T23:07:02.8959458Z 2024-08-20T23:07:02.8960190Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:02.8960853Z cuda train ElectraForCausalLM 2024-08-20T23:07:37.1650662Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:07:37.1654978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:07:37.1656203Z pred = mod(**cloned_inputs) 2024-08-20T23:07:37.1657667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1617, in forward 2024-08-20T23:07:37.1658590Z outputs = self.electra( 2024-08-20T23:07:37.1659577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-08-20T23:07:37.1660799Z hidden_states = self.embeddings( 2024-08-20T23:07:37.1662639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-08-20T23:07:37.1663943Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:07:37.1664364Z 2024-08-20T23:07:37.9636462Z pass 2024-08-20T23:07:37.9827641Z TIMING: entire_frame_compile:22.51406 code_gen:9.78915 inductor_compile:17.1289 backend_compile:16.99574 total_wall_time:32.32461 2024-08-20T23:07:37.9829390Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:40391 | ProxyTorchDispatchMode.__torch_dispatch__:17991 | FakeTensor.__torch_dispatch__:6838 2024-08-20T23:07:37.9830503Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-08-20T23:07:43.3998401Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:07:43.4000135Z warnings.warn( 2024-08-20T23:07:43.6067991Z 2024-08-20T23:07:44.4209716Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:44.4210471Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:44.4210974Z cuda train ElectraForQuestionAnswering 2024-08-20T23:08:18.1799939Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:08:18.1803186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:08:18.1804363Z pred = mod(**cloned_inputs) 2024-08-20T23:08:18.1805575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1385, in forward 2024-08-20T23:08:18.1806845Z discriminator_hidden_states = self.electra( 2024-08-20T23:08:18.1808152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-08-20T23:08:18.1809461Z hidden_states = self.embeddings( 2024-08-20T23:08:18.1810658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-08-20T23:08:18.1811632Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:08:18.1811978Z 2024-08-20T23:08:18.3084954Z W0820 23:08:18.307000 13806 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:09:05.0911235Z pass 2024-08-20T23:09:05.1058280Z TIMING: entire_frame_compile:66.65521 code_gen:25.22548 inductor_compile:42.51048 backend_compile:52.5656 total_wall_time:76.26636 2024-08-20T23:09:05.1059673Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:60931 | ProxyTorchDispatchMode.__torch_dispatch__:26964 | FakeTensor.__torch_dispatch__:16890 2024-08-20T23:09:05.1065198Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T23:09:12.5015314Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:09:12.5016942Z warnings.warn( 2024-08-20T23:09:12.7075166Z 2024-08-20T23:09:15.1737988Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:09:15.1738492Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:09:15.1738967Z cuda train GPT2ForSequenceClassification 2024-08-20T23:09:52.0701852Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:09:52.0703456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-08-20T23:09:52.0704465Z transformer_outputs = self.transformer( 2024-08-20T23:09:52.0710643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-20T23:09:52.0711568Z inputs_embeds = self.wte(input_ids) 2024-08-20T23:09:52.0711861Z 2024-08-20T23:09:52.1413426Z W0820 23:09:52.140000 14030 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:10:28.2338754Z pass 2024-08-20T23:10:28.3133830Z TIMING: entire_frame_compile:55.03472 inductor_compile:35.7773 backend_compile:42.66902 code_gen:21.87608 total_wall_time:64.28731 2024-08-20T23:10:28.3135716Z STATS: call_* op count: 1101 | FakeTensorMode.__torch_dispatch__:55263 | ProxyTorchDispatchMode.__torch_dispatch__:22507 | FakeTensor.__torch_dispatch__:13573 2024-08-20T23:10:28.3136841Z Dynamo produced 5 graphs covering 1101 ops with 7 graph breaks (5 unique) 2024-08-20T23:10:35.0542754Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:10:35.0544406Z warnings.warn( 2024-08-20T23:10:35.2444588Z 2024-08-20T23:10:36.7617726Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:36.9785552Z 2024-08-20T23:10:36.9786170Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:10:36.9786720Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:36.9796002Z cuda train GoogleFnet 2024-08-20T23:10:38.3192277Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:56.8816059Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:10:56.8817387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:10:56.8818364Z pred = mod(**cloned_inputs) 2024-08-20T23:10:56.8820277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 749, in forward 2024-08-20T23:10:56.8821166Z outputs = self.fnet( 2024-08-20T23:10:56.8822102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 581, in forward 2024-08-20T23:10:56.8822992Z embedding_output = self.embeddings( 2024-08-20T23:10:56.8823996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 148, in forward 2024-08-20T23:10:56.8824948Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:10:56.8825334Z 2024-08-20T23:10:56.9925600Z W0820 23:10:56.991000 14341 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:11:21.9418322Z pass 2024-08-20T23:11:21.9843973Z TIMING: entire_frame_compile:34.89894 code_gen:15.14638 inductor_compile:25.11907 backend_compile:27.64571 total_wall_time:40.8435 2024-08-20T23:11:21.9846756Z STATS: call_* op count: 573 | FakeTensorMode.__torch_dispatch__:27076 | ProxyTorchDispatchMode.__torch_dispatch__:12285 | FakeTensor.__torch_dispatch__:8814 2024-08-20T23:11:21.9849300Z Dynamo produced 2 graphs covering 573 ops with 5 graph breaks (4 unique) 2024-08-20T23:11:27.6230440Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:11:27.6232104Z warnings.warn( 2024-08-20T23:11:27.8508327Z 2024-08-20T23:11:30.0452438Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:30.0452915Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:11:30.0453364Z cuda train LayoutLMForMaskedLM 2024-08-20T23:12:06.8426279Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:12:06.8428127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:12:06.8429374Z pred = mod(**cloned_inputs) 2024-08-20T23:12:06.8430931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 938, in forward 2024-08-20T23:12:06.8432210Z outputs = self.layoutlm( 2024-08-20T23:12:06.8433295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-08-20T23:12:06.8434703Z embedding_output = self.embeddings( 2024-08-20T23:12:06.8435928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-08-20T23:12:06.8436901Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:12:06.8437249Z 2024-08-20T23:12:06.9821858Z W0820 23:12:06.981000 14660 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:12:53.3591132Z pass 2024-08-20T23:12:53.4241258Z TIMING: entire_frame_compile:66.68893 code_gen:26.25414 inductor_compile:43.88172 backend_compile:53.18443 total_wall_time:76.86652 2024-08-20T23:12:53.4242774Z STATS: call_* op count: 1033 | FakeTensorMode.__torch_dispatch__:62180 | ProxyTorchDispatchMode.__torch_dispatch__:27526 | FakeTensor.__torch_dispatch__:17218 2024-08-20T23:12:53.4244071Z Dynamo produced 2 graphs covering 1033 ops with 5 graph breaks (4 unique) 2024-08-20T23:13:00.9762287Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:13:00.9763911Z warnings.warn( 2024-08-20T23:13:01.1926613Z 2024-08-20T23:13:03.1304416Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:13:03.1304886Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:13:03.1305391Z cuda train LayoutLMForSequenceClassification 2024-08-20T23:13:41.0192319Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:13:41.0195392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-08-20T23:13:41.0196353Z outputs = self.layoutlm( 2024-08-20T23:13:41.0197361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-08-20T23:13:41.0198326Z embedding_output = self.embeddings( 2024-08-20T23:13:41.0199380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-08-20T23:13:41.0200356Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:13:41.0200703Z 2024-08-20T23:13:41.1586395Z W0820 23:13:41.157000 14862 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:14:28.6140468Z pass 2024-08-20T23:14:28.6841809Z TIMING: entire_frame_compile:68.50298 inductor_compile:42.06499 backend_compile:52.83383 code_gen:23.31041 total_wall_time:77.75095 2024-08-20T23:14:28.6845582Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:66596 | ProxyTorchDispatchMode.__torch_dispatch__:27800 | FakeTensor.__torch_dispatch__:16804 2024-08-20T23:14:28.6847700Z Dynamo produced 5 graphs covering 1030 ops with 7 graph breaks (5 unique) 2024-08-20T23:14:35.9829373Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:14:35.9831067Z warnings.warn( 2024-08-20T23:14:36.2036699Z 2024-08-20T23:14:47.0176627Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:14:47.0177095Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:14:47.0178156Z cuda train M2M100ForConditionalGeneration 2024-08-20T23:14:47.0944617Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:16:04.1550776Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:16:04.1552502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:16:04.1554500Z pred = mod(**cloned_inputs) 2024-08-20T23:16:04.1556185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1275, in forward 2024-08-20T23:16:04.1557512Z outputs = self.model( 2024-08-20T23:16:04.1558977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1162, in forward 2024-08-20T23:16:04.1560526Z encoder_outputs = self.encoder( 2024-08-20T23:16:04.1562284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 763, in forward 2024-08-20T23:16:04.1564071Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-08-20T23:16:04.1564801Z 2024-08-20T23:16:05.9887450Z pass 2024-08-20T23:16:06.0823067Z TIMING: entire_frame_compile:53.0408 code_gen:21.7017 inductor_compile:40.65585 backend_compile:40.51871 total_wall_time:74.21222 2024-08-20T23:16:06.0825884Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:101924 | ProxyTorchDispatchMode.__torch_dispatch__:47007 | FakeTensor.__torch_dispatch__:14711 2024-08-20T23:16:06.0827167Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-08-20T23:16:13.4716675Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:16:13.4718353Z warnings.warn( 2024-08-20T23:16:13.6688099Z 2024-08-20T23:16:17.7409296Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:17.7409774Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:16:17.7410331Z cuda train MBartForCausalLM 2024-08-20T23:16:17.7722453Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-08-20T23:16:25.2899216Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:16:25.2900618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T23:16:25.2901563Z return super().forward(positions + self.offset) 2024-08-20T23:16:25.2901923Z 2024-08-20T23:16:31.0394110Z pass 2024-08-20T23:16:31.1023718Z TIMING: entire_frame_compile:4.47794 inductor_compile:3.74974 backend_compile:3.62137 code_gen:1.95282 total_wall_time:5.74588 2024-08-20T23:16:31.1025121Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:4540 | ProxyTorchDispatchMode.__torch_dispatch__:1925 | FakeTensor.__torch_dispatch__:655 2024-08-20T23:16:31.1026239Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:16:35.2181943Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:16:35.2183607Z warnings.warn( 2024-08-20T23:16:35.4582081Z 2024-08-20T23:16:42.6170438Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:42.6170909Z loading model: 0it [00:07, ?it/s] 2024-08-20T23:16:42.6171375Z cuda train MBartForConditionalGeneration 2024-08-20T23:16:42.7155728Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:16:55.5804387Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:16:55.5805808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T23:16:55.5806749Z return super().forward(positions + self.offset) 2024-08-20T23:16:55.5807098Z 2024-08-20T23:17:10.8734672Z pass 2024-08-20T23:17:10.9786371Z TIMING: entire_frame_compile:7.55761 inductor_compile:6.20187 backend_compile:5.95069 code_gen:3.18426 total_wall_time:9.98342 2024-08-20T23:17:10.9788180Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:10227 | ProxyTorchDispatchMode.__torch_dispatch__:4496 | FakeTensor.__torch_dispatch__:1625 2024-08-20T23:17:10.9789296Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique) 2024-08-20T23:17:15.3185896Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:17:15.3187654Z warnings.warn( 2024-08-20T23:17:15.5378028Z 2024-08-20T23:17:19.7441482Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:17:19.7442081Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:17:19.7442633Z cuda train MT5ForConditionalGeneration 2024-08-20T23:18:15.1139758Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:18:15.1142400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:18:15.1143401Z pred = mod(**cloned_inputs) 2024-08-20T23:18:15.1144618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1722, in forward 2024-08-20T23:18:15.1147368Z encoder_outputs = self.encoder( 2024-08-20T23:18:15.1148367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 989, in forward 2024-08-20T23:18:15.1149288Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:18:15.1149622Z 2024-08-20T23:18:15.2461339Z W0820 23:18:15.245000 15729 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:19:02.9554521Z pass 2024-08-20T23:19:03.0989377Z TIMING: entire_frame_compile:79.90074 code_gen:29.01992 inductor_compile:52.95787 backend_compile:61.95113 total_wall_time:96.17749 2024-08-20T23:19:03.0990845Z STATS: call_* op count: 1763 | FakeTensorMode.__torch_dispatch__:97878 | ProxyTorchDispatchMode.__torch_dispatch__:44564 | FakeTensor.__torch_dispatch__:21452 2024-08-20T23:19:03.0991981Z Dynamo produced 2 graphs covering 1763 ops with 5 graph breaks (4 unique) 2024-08-20T23:19:11.2015171Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:19:11.2016834Z warnings.warn( 2024-08-20T23:19:11.4106007Z 2024-08-20T23:19:11.4144440Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T23:19:16.2667925Z 2024-08-20T23:19:16.2668637Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:19:16.2669253Z cuda train MegatronBertForCausalLM 2024-08-20T23:20:24.3061842Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:20:24.3064451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:20:24.3065533Z pred = mod(**cloned_inputs) 2024-08-20T23:20:24.3066941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1199, in forward 2024-08-20T23:20:24.3068486Z outputs = self.bert( 2024-08-20T23:20:24.3069698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-08-20T23:20:24.3070769Z embedding_output = self.embeddings( 2024-08-20T23:20:24.3072214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-08-20T23:20:24.3073567Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:20:24.3073919Z 2024-08-20T23:20:24.4945969Z W0820 23:20:24.493000 16123 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:21:57.9343477Z pass 2024-08-20T23:21:58.1352437Z TIMING: entire_frame_compile:129.63746 code_gen:50.55369 inductor_compile:84.45015 backend_compile:102.85694 total_wall_time:147.75164 2024-08-20T23:21:58.1354769Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:118255 | ProxyTorchDispatchMode.__torch_dispatch__:52648 | FakeTensor.__torch_dispatch__:32930 2024-08-20T23:21:58.1356083Z Dynamo produced 2 graphs covering 1922 ops with 5 graph breaks (4 unique) 2024-08-20T23:22:08.8863839Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:22:08.8865517Z warnings.warn( 2024-08-20T23:22:09.0775316Z 2024-08-20T23:22:13.6546380Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:13.6546861Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:22:13.6547333Z cuda train MegatronBertForQuestionAnswering 2024-08-20T23:23:21.1101846Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:23:21.1103533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:23:21.1104502Z pred = mod(**cloned_inputs) 2024-08-20T23:23:21.1106603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1792, in forward 2024-08-20T23:23:21.1107657Z outputs = self.bert( 2024-08-20T23:23:21.1108837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-08-20T23:23:21.1109926Z embedding_output = self.embeddings( 2024-08-20T23:23:21.1111108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-08-20T23:23:21.1112246Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:23:21.1112616Z 2024-08-20T23:23:21.2863988Z W0820 23:23:21.285000 16538 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:24:48.2143580Z pass 2024-08-20T23:24:48.4150693Z TIMING: entire_frame_compile:123.10366 code_gen:43.8494 inductor_compile:77.37193 backend_compile:96.63521 total_wall_time:140.94503 2024-08-20T23:24:48.4153185Z STATS: call_* op count: 1916 | FakeTensorMode.__torch_dispatch__:117758 | ProxyTorchDispatchMode.__torch_dispatch__:52417 | FakeTensor.__torch_dispatch__:32732 2024-08-20T23:24:48.4154360Z Dynamo produced 2 graphs covering 1916 ops with 5 graph breaks (4 unique) 2024-08-20T23:24:59.3148534Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:24:59.3150283Z warnings.warn( 2024-08-20T23:24:59.5141194Z 2024-08-20T23:25:00.9762286Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:00.9762800Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:25:00.9763834Z cuda train MobileBertForMaskedLM 2024-08-20T23:27:04.3312097Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:27:04.3313283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:27:04.3316891Z pred = mod(**cloned_inputs) 2024-08-20T23:27:04.3319181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1089, in forward 2024-08-20T23:27:04.3320759Z outputs = self.mobilebert( 2024-08-20T23:27:04.3321902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-08-20T23:27:04.3322946Z embedding_output = self.embeddings( 2024-08-20T23:27:04.3324177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-08-20T23:27:04.3325274Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:27:04.3325633Z 2024-08-20T23:27:06.9392905Z pass 2024-08-20T23:27:06.9874952Z TIMING: entire_frame_compile:86.12943 code_gen:31.33984 inductor_compile:57.87366 backend_compile:59.22697 total_wall_time:119.36265 2024-08-20T23:27:06.9876553Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:158034 | ProxyTorchDispatchMode.__torch_dispatch__:74465 | FakeTensor.__torch_dispatch__:23011 2024-08-20T23:27:06.9877710Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-08-20T23:27:15.9424956Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:27:15.9426599Z warnings.warn( 2024-08-20T23:27:16.1786381Z 2024-08-20T23:27:17.5045673Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:27:17.5046704Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:27:17.5047609Z cuda train MobileBertForQuestionAnswering 2024-08-20T23:29:19.3849210Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:29:19.3850694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:29:19.3851799Z pred = mod(**cloned_inputs) 2024-08-20T23:29:19.3854090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1390, in forward 2024-08-20T23:29:19.3855291Z outputs = self.mobilebert( 2024-08-20T23:29:19.3856655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-08-20T23:29:19.3857860Z embedding_output = self.embeddings( 2024-08-20T23:29:19.3859004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-08-20T23:29:19.3860033Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:29:19.3860400Z 2024-08-20T23:29:22.0094313Z pass 2024-08-20T23:29:22.0435492Z TIMING: entire_frame_compile:85.3188 code_gen:30.51736 inductor_compile:56.68201 backend_compile:58.43423 total_wall_time:117.9787 2024-08-20T23:29:22.0438499Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:157881 | ProxyTorchDispatchMode.__torch_dispatch__:74369 | FakeTensor.__torch_dispatch__:22989 2024-08-20T23:29:22.0439671Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-08-20T23:29:31.4723094Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:29:31.4724772Z warnings.warn( 2024-08-20T23:29:31.6818071Z 2024-08-20T23:29:34.3404643Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:34.3405353Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:29:34.3406074Z cuda train OPTForCausalLM 2024-08-20T23:29:34.3600485Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-08-20T23:29:43.5125143Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:29:43.5126930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 109, in forward 2024-08-20T23:29:43.5127872Z return super().forward(positions + self.offset) 2024-08-20T23:29:43.5128221Z 2024-08-20T23:29:49.4024418Z pass 2024-08-20T23:29:49.4278921Z TIMING: entire_frame_compile:4.96588 inductor_compile:4.69616 backend_compile:4.07625 code_gen:3.12841 total_wall_time:6.78519 2024-08-20T23:29:49.4280347Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:5003 | ProxyTorchDispatchMode.__torch_dispatch__:2038 | FakeTensor.__torch_dispatch__:802 2024-08-20T23:29:49.4281463Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique) 2024-08-20T23:29:53.6089504Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:29:53.6091272Z warnings.warn( 2024-08-20T23:29:53.8052839Z 2024-08-20T23:29:55.7411099Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:55.7411707Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:29:55.7412229Z cuda train PLBartForCausalLM 2024-08-20T23:29:55.8522846Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-08-20T23:30:04.8114575Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:30:04.8116246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-08-20T23:30:04.8117221Z return super().forward(positions + self.offset) 2024-08-20T23:30:04.8117572Z 2024-08-20T23:30:08.1212339Z pass 2024-08-20T23:30:08.1508290Z TIMING: entire_frame_compile:5.63178 inductor_compile:4.84046 backend_compile:4.76707 code_gen:2.61692 total_wall_time:7.56859 2024-08-20T23:30:08.1509822Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:4751 | ProxyTorchDispatchMode.__torch_dispatch__:1942 | FakeTensor.__torch_dispatch__:796 2024-08-20T23:30:08.1510935Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:30:12.3077312Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:30:12.3079951Z warnings.warn( 2024-08-20T23:30:12.5192964Z 2024-08-20T23:30:16.0461921Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:16.0462970Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:30:16.0463893Z cuda train PLBartForConditionalGeneration 2024-08-20T23:30:16.0992107Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:30:28.1354087Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:30:28.1356716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-08-20T23:30:28.1358527Z return super().forward(positions + self.offset) 2024-08-20T23:30:28.1358873Z 2024-08-20T23:30:36.7241492Z pass 2024-08-20T23:30:36.7626173Z TIMING: entire_frame_compile:7.88987 inductor_compile:7.00611 backend_compile:6.25478 code_gen:3.64221 total_wall_time:11.04943 2024-08-20T23:30:36.7628170Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:10524 | ProxyTorchDispatchMode.__torch_dispatch__:4595 | FakeTensor.__torch_dispatch__:1763 2024-08-20T23:30:36.7629558Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique) 2024-08-20T23:30:41.1264442Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:30:41.1266588Z warnings.warn( 2024-08-20T23:30:41.3384078Z 2024-08-20T23:30:47.5199631Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:47.5200649Z loading model: 0it [00:06, ?it/s] 2024-08-20T23:30:47.5201541Z cuda train PegasusForCausalLM 2024-08-20T23:30:47.5511810Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-08-20T23:31:02.2211074Z pass 2024-08-20T23:31:02.2975076Z TIMING: entire_frame_compile:5.52699 inductor_compile:4.69654 backend_compile:4.69976 code_gen:2.42536 total_wall_time:7.38553 2024-08-20T23:31:02.2977816Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4616 | ProxyTorchDispatchMode.__torch_dispatch__:1907 | FakeTensor.__torch_dispatch__:793 2024-08-20T23:31:02.2979152Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T23:31:06.4797054Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:31:06.4799600Z warnings.warn( 2024-08-20T23:31:06.6922193Z 2024-08-20T23:31:16.9845495Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:16.9846180Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:31:16.9846889Z cuda train PegasusForConditionalGeneration 2024-08-20T23:31:17.0605301Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:31:43.9939561Z pass 2024-08-20T23:31:44.1212817Z TIMING: entire_frame_compile:7.39419 inductor_compile:6.24259 backend_compile:5.8247 code_gen:2.94631 total_wall_time:10.12085 2024-08-20T23:31:44.1214519Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9942 | ProxyTorchDispatchMode.__torch_dispatch__:4369 | FakeTensor.__torch_dispatch__:1731 2024-08-20T23:31:44.1215748Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique) 2024-08-20T23:31:48.5182423Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:31:48.5184124Z warnings.warn( 2024-08-20T23:31:48.7326159Z 2024-08-20T23:31:48.7355479Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-08-20T23:31:50.8949754Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T23:31:50.8952391Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T23:31:51.1922193Z 2024-08-20T23:31:51.1922748Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:31:51.1923507Z cuda train RobertaForCausalLM 2024-08-20T23:32:26.8521073Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:32:26.8522268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:32:26.8525882Z pred = mod(**cloned_inputs) 2024-08-20T23:32:26.8527330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 953, in forward 2024-08-20T23:32:26.8528325Z outputs = self.roberta( 2024-08-20T23:32:26.8529411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-08-20T23:32:26.8530606Z embedding_output = self.embeddings( 2024-08-20T23:32:26.8532061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-08-20T23:32:26.8533160Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:32:26.8533511Z 2024-08-20T23:32:26.9878343Z W0820 23:32:26.987000 18199 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:33:12.2596621Z pass 2024-08-20T23:33:12.3132745Z TIMING: entire_frame_compile:64.95194 code_gen:23.74691 inductor_compile:42.02323 backend_compile:51.75064 total_wall_time:74.60452 2024-08-20T23:33:12.3134931Z STATS: call_* op count: 1007 | FakeTensorMode.__torch_dispatch__:61027 | ProxyTorchDispatchMode.__torch_dispatch__:27046 | FakeTensor.__torch_dispatch__:16964 2024-08-20T23:33:12.3136671Z Dynamo produced 2 graphs covering 1007 ops with 5 graph breaks (4 unique) 2024-08-20T23:33:19.6588275Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:33:19.6590022Z warnings.warn( 2024-08-20T23:33:19.8542924Z 2024-08-20T23:33:21.6098226Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T23:33:21.6100424Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T23:33:21.9452426Z 2024-08-20T23:33:21.9452704Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:33:21.9457995Z cuda train RobertaForQuestionAnswering 2024-08-20T23:33:56.1170223Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:33:56.1172294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:33:56.1174013Z pred = mod(**cloned_inputs) 2024-08-20T23:33:56.1177243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1500, in forward 2024-08-20T23:33:56.1178183Z outputs = self.roberta( 2024-08-20T23:33:56.1179233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-08-20T23:33:56.1180244Z embedding_output = self.embeddings( 2024-08-20T23:33:56.1181278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-08-20T23:33:56.1182238Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:33:56.1182593Z 2024-08-20T23:33:56.2275716Z W0820 23:33:56.226000 18377 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:34:38.4642071Z pass 2024-08-20T23:34:38.5298677Z TIMING: entire_frame_compile:61.32945 code_gen:20.74849 inductor_compile:37.84549 backend_compile:47.39088 total_wall_time:70.64836 2024-08-20T23:34:38.5300169Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:60506 | ProxyTorchDispatchMode.__torch_dispatch__:26802 | FakeTensor.__torch_dispatch__:16758 2024-08-20T23:34:38.5302297Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T23:34:45.8081469Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:34:45.8084046Z warnings.warn( 2024-08-20T23:34:45.9963261Z 2024-08-20T23:34:46.6496139Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:46.6496736Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:46.6497292Z cuda train Speech2Text2ForCausalLM 2024-08-20T23:34:46.7602655Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-08-20T23:34:57.6105211Z pass 2024-08-20T23:34:57.6156688Z TIMING: entire_frame_compile:4.8886 inductor_compile:4.50961 backend_compile:4.03055 code_gen:2.35858 total_wall_time:6.4896 2024-08-20T23:34:57.6158797Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:4770 | ProxyTorchDispatchMode.__torch_dispatch__:1940 | FakeTensor.__torch_dispatch__:739 2024-08-20T23:34:57.6160498Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique) 2024-08-20T23:35:01.7862111Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:35:01.7863780Z warnings.warn( 2024-08-20T23:35:02.0154293Z 2024-08-20T23:35:03.5613236Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:35:03.5613727Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:35:03.5616042Z cuda train T5ForConditionalGeneration 2024-08-20T23:35:43.7696834Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:35:43.7698047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:35:43.7699001Z pred = mod(**cloned_inputs) 2024-08-20T23:35:43.7700863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-08-20T23:35:43.7702027Z encoder_outputs = self.encoder( 2024-08-20T23:35:43.7703327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-08-20T23:35:43.7704249Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:35:43.7704584Z 2024-08-20T23:35:43.8748667Z W0820 23:35:43.873000 18758 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:36:17.0656234Z pass 2024-08-20T23:36:17.1233662Z TIMING: entire_frame_compile:56.32893 code_gen:20.39264 inductor_compile:35.42185 backend_compile:43.30568 total_wall_time:66.52639 2024-08-20T23:36:17.1236755Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:67853 | ProxyTorchDispatchMode.__torch_dispatch__:30411 | FakeTensor.__torch_dispatch__:14024 2024-08-20T23:36:17.1237920Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T23:36:23.7621468Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:36:23.7623349Z warnings.warn( 2024-08-20T23:36:23.9551355Z 2024-08-20T23:36:25.5159196Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:36:25.5159669Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:36:25.5162712Z cuda train T5Small 2024-08-20T23:36:50.0317062Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:36:50.0318797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:36:50.0320641Z pred = mod(**cloned_inputs) 2024-08-20T23:36:50.0321992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-08-20T23:36:50.0322902Z encoder_outputs = self.encoder( 2024-08-20T23:36:50.0326340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-08-20T23:36:50.0327785Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:36:50.0328133Z 2024-08-20T23:36:50.1164346Z W0820 23:36:50.115000 19111 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:37:06.8852885Z pass 2024-08-20T23:37:06.9427798Z TIMING: entire_frame_compile:33.8826 inductor_compile:5.13616 backend_compile:21.64102 total_wall_time:35.19496 2024-08-20T23:37:06.9429221Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:55256 | ProxyTorchDispatchMode.__torch_dispatch__:30411 | FakeTensor.__torch_dispatch__:2452 2024-08-20T23:37:06.9430403Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T23:37:11.5863404Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:37:11.5865054Z warnings.warn( 2024-08-20T23:37:11.7649687Z 2024-08-20T23:37:15.8225595Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:15.8226082Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:37:15.8226525Z cuda train TrOCRForCausalLM 2024-08-20T23:37:15.8530026Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-08-20T23:37:24.6765708Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:37:24.6767337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 66, in forward 2024-08-20T23:37:24.6768302Z return super().forward(positions + self.offset) 2024-08-20T23:37:24.6768654Z 2024-08-20T23:37:30.5059070Z pass 2024-08-20T23:37:30.5861018Z TIMING: entire_frame_compile:5.47558 inductor_compile:4.37397 backend_compile:4.60141 code_gen:2.84413 total_wall_time:7.24745 2024-08-20T23:37:30.5863074Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:4709 | ProxyTorchDispatchMode.__torch_dispatch__:1925 | FakeTensor.__torch_dispatch__:796 2024-08-20T23:37:30.5864763Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-08-20T23:37:34.7820019Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:37:34.7822616Z warnings.warn( 2024-08-20T23:37:34.9641580Z 2024-08-20T23:37:44.9956163Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:44.9956673Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:37:44.9958499Z cuda train XGLMForCausalLM 2024-08-20T23:37:45.0490670Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-08-20T23:37:49.3284974Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 2024-08-20T23:37:49.3286425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 174, in forward 2024-08-20T23:37:49.3287361Z position_ids += self.offset 2024-08-20T23:37:49.3287623Z 2024-08-20T23:38:05.8129105Z pass 2024-08-20T23:38:05.9028443Z TIMING: entire_frame_compile:5.19784 inductor_compile:4.66818 backend_compile:4.16962 code_gen:2.54739 total_wall_time:6.89733 2024-08-20T23:38:05.9030269Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:4948 | ProxyTorchDispatchMode.__torch_dispatch__:2016 | FakeTensor.__torch_dispatch__:835 2024-08-20T23:38:05.9031381Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique) 2024-08-20T23:38:10.1414554Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:38:10.1416638Z warnings.warn( 2024-08-20T23:38:10.3446653Z 2024-08-20T23:38:15.0583624Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:15.0584229Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:38:15.0585897Z cuda train XLNetLMHeadModel 2024-08-20T23:39:05.5880743Z W0820 23:39:05.587000 19690 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-20T23:39:23.2372539Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-08-20T23:39:23.2374480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:39:23.2375559Z pred = mod(**cloned_inputs) 2024-08-20T23:39:23.2379364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1446, in forward 2024-08-20T23:39:23.2380623Z transformer_outputs = self.transformer( 2024-08-20T23:39:23.2381864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1203, in forward 2024-08-20T23:39:23.2382887Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2024-08-20T23:39:23.2384151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1055, in relative_positional_encoding 2024-08-20T23:39:23.2385276Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2024-08-20T23:39:23.2386488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1014, in positional_embedding 2024-08-20T23:39:23.2387455Z pos_emb = pos_emb[:, None, :] 2024-08-20T23:39:23.2387723Z 2024-08-20T23:39:48.7157691Z W0820 23:39:48.714000 19690 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:41:18.1707267Z pass 2024-08-20T23:41:18.3830982Z TIMING: entire_frame_compile:147.15163 code_gen:52.50009 inductor_compile:91.78267 backend_compile:115.55231 total_wall_time:169.09668 2024-08-20T23:41:18.3835890Z STATS: call_* op count: 1873 | FakeTensorMode.__torch_dispatch__:162756 | ProxyTorchDispatchMode.__torch_dispatch__:66725 | FakeTensor.__torch_dispatch__:33630 2024-08-20T23:41:18.3837891Z Dynamo produced 2 graphs covering 1873 ops with 5 graph breaks (4 unique) 2024-08-20T23:41:29.5245574Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:41:29.5247304Z warnings.warn( 2024-08-20T23:41:29.7513833Z 2024-08-20T23:41:32.0742759Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:32.0743284Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:32.0746542Z cuda train YituTechConvBert 2024-08-20T23:42:26.5741047Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:42:26.5742402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:42:26.5743546Z pred = mod(**cloned_inputs) 2024-08-20T23:42:26.5749194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 918, in forward 2024-08-20T23:42:26.5750799Z generator_hidden_states = self.convbert( 2024-08-20T23:42:26.5751892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 834, in forward 2024-08-20T23:42:26.5752834Z hidden_states = self.embeddings( 2024-08-20T23:42:26.5753881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 230, in forward 2024-08-20T23:42:26.5755071Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:42:26.5755414Z 2024-08-20T23:42:26.7447629Z W0820 23:42:26.743000 20116 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:43:33.7368682Z pass 2024-08-20T23:43:33.7905689Z TIMING: entire_frame_compile:98.25334 code_gen:37.49102 inductor_compile:63.70572 backend_compile:77.59737 total_wall_time:113.88485 2024-08-20T23:43:33.7908268Z STATS: call_* op count: 1511 | FakeTensorMode.__torch_dispatch__:90824 | ProxyTorchDispatchMode.__torch_dispatch__:39917 | FakeTensor.__torch_dispatch__:24103 2024-08-20T23:43:33.7909470Z Dynamo produced 2 graphs covering 1511 ops with 5 graph breaks (4 unique) 2024-08-20T23:43:39.9334022Z accuracy pass_rate=93.48% 2024-08-20T23:43:39.9337340Z calls_captured gmean=0.00x mean=750.522x 2024-08-20T23:43:39.9340787Z unique_graphs gmean=0.00x mean=3.348x 2024-08-20T23:43:39.9343796Z graph_breaks gmean=0.00x mean=5.196x 2024-08-20T23:43:39.9347121Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-08-20T23:43:39.9350489Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:43:39.9353826Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:43:39.9357278Z cudagraph_skips gmean=0.00x mean=0.891x 2024-08-20T23:43:41.0999948Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_huggingface_training.csv 2024-08-20T23:43:41.3614523Z AlbertForMaskedLM PASS 2024-08-20T23:43:41.3619907Z AlbertForQuestionAnswering PASS 2024-08-20T23:43:41.3625075Z AllenaiLongformerBase PASS 2024-08-20T23:43:41.3630199Z BartForCausalLM PASS 2024-08-20T23:43:41.3635520Z BartForConditionalGeneration PASS 2024-08-20T23:43:41.3640546Z BertForMaskedLM PASS 2024-08-20T23:43:41.3645772Z BertForQuestionAnswering PASS 2024-08-20T23:43:41.3650643Z BlenderbotForCausalLM XFAIL 2024-08-20T23:43:41.3656114Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:43:41.3661145Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:43:41.3666347Z CamemBert PASS 2024-08-20T23:43:41.3671397Z DebertaForMaskedLM PASS 2024-08-20T23:43:41.3676403Z DebertaForQuestionAnswering PASS 2024-08-20T23:43:41.3681309Z DebertaV2ForMaskedLM XFAIL 2024-08-20T23:43:41.3686325Z DebertaV2ForQuestionAnswering XFAIL 2024-08-20T23:43:41.3691618Z DistilBertForMaskedLM PASS 2024-08-20T23:43:41.3696742Z DistilBertForQuestionAnswering PASS 2024-08-20T23:43:41.3701669Z DistillGPT2 PASS 2024-08-20T23:43:41.3706672Z ElectraForCausalLM PASS 2024-08-20T23:43:41.3711718Z ElectraForQuestionAnswering PASS 2024-08-20T23:43:41.3717506Z GPT2ForSequenceClassification PASS 2024-08-20T23:43:41.3722412Z GoogleFnet PASS 2024-08-20T23:43:41.3727340Z LayoutLMForMaskedLM PASS 2024-08-20T23:43:41.3732680Z LayoutLMForSequenceClassification PASS 2024-08-20T23:43:41.3737615Z M2M100ForConditionalGeneration PASS 2024-08-20T23:43:41.3742704Z MBartForCausalLM PASS 2024-08-20T23:43:41.3747673Z MBartForConditionalGeneration PASS 2024-08-20T23:43:41.3752731Z MT5ForConditionalGeneration PASS 2024-08-20T23:43:41.3757764Z MegatronBertForCausalLM PASS 2024-08-20T23:43:41.3763874Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:43:41.3767732Z MobileBertForMaskedLM PASS 2024-08-20T23:43:41.3773082Z MobileBertForQuestionAnswering PASS 2024-08-20T23:43:41.3777870Z OPTForCausalLM PASS 2024-08-20T23:43:41.3783028Z PLBartForCausalLM PASS 2024-08-20T23:43:41.3788215Z PLBartForConditionalGeneration PASS 2024-08-20T23:43:41.3793221Z PegasusForCausalLM PASS 2024-08-20T23:43:41.3798204Z PegasusForConditionalGeneration PASS 2024-08-20T23:43:41.3803177Z RobertaForCausalLM PASS 2024-08-20T23:43:41.3808169Z RobertaForQuestionAnswering PASS 2024-08-20T23:43:41.3813420Z Speech2Text2ForCausalLM PASS 2024-08-20T23:43:41.3818946Z T5ForConditionalGeneration PASS 2024-08-20T23:43:41.3823886Z T5Small PASS 2024-08-20T23:43:41.3828790Z TrOCRForCausalLM PASS 2024-08-20T23:43:41.3833883Z XGLMForCausalLM PASS 2024-08-20T23:43:41.3838943Z XLNetLMHeadModel PASS 2024-08-20T23:43:41.3843941Z YituTechConvBert PASS 2024-08-20T23:43:41.4270631Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_huggingface_training.csv 2024-08-20T23:43:41.6910218Z AlbertForMaskedLM PASS 2024-08-20T23:43:41.6914266Z AlbertForQuestionAnswering PASS 2024-08-20T23:43:41.6919710Z AllenaiLongformerBase PASS 2024-08-20T23:43:41.6924994Z BartForCausalLM PASS 2024-08-20T23:43:41.6930318Z BartForConditionalGeneration PASS 2024-08-20T23:43:41.6935650Z BertForMaskedLM PASS 2024-08-20T23:43:41.6941380Z BertForQuestionAnswering PASS 2024-08-20T23:43:41.6946384Z BlenderbotForCausalLM PASS 2024-08-20T23:43:41.6951729Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:43:41.6957033Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:43:41.6962212Z CamemBert PASS 2024-08-20T23:43:41.6967325Z DebertaForMaskedLM PASS 2024-08-20T23:43:41.6972744Z DebertaForQuestionAnswering PASS 2024-08-20T23:43:41.6977788Z DebertaV2ForMaskedLM PASS 2024-08-20T23:43:41.6983007Z DebertaV2ForQuestionAnswering PASS 2024-08-20T23:43:41.6988151Z DistilBertForMaskedLM PASS 2024-08-20T23:43:41.6993288Z DistilBertForQuestionAnswering PASS 2024-08-20T23:43:41.6998261Z DistillGPT2 PASS 2024-08-20T23:43:41.7003430Z ElectraForCausalLM PASS 2024-08-20T23:43:41.7008617Z ElectraForQuestionAnswering PASS 2024-08-20T23:43:41.7014185Z GPT2ForSequenceClassification PASS 2024-08-20T23:43:41.7019618Z GoogleFnet PASS 2024-08-20T23:43:41.7024819Z LayoutLMForMaskedLM PASS 2024-08-20T23:43:41.7029916Z LayoutLMForSequenceClassification PASS 2024-08-20T23:43:41.7035094Z M2M100ForConditionalGeneration PASS 2024-08-20T23:43:41.7040357Z MBartForCausalLM PASS 2024-08-20T23:43:41.7045579Z MBartForConditionalGeneration PASS 2024-08-20T23:43:41.7050788Z MT5ForConditionalGeneration PASS 2024-08-20T23:43:41.7056189Z MegatronBertForCausalLM PASS 2024-08-20T23:43:41.7061326Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:43:41.7066493Z MobileBertForMaskedLM PASS 2024-08-20T23:43:41.7071510Z MobileBertForQuestionAnswering PASS 2024-08-20T23:43:41.7076650Z OPTForCausalLM PASS 2024-08-20T23:43:41.7081787Z PLBartForCausalLM PASS 2024-08-20T23:43:41.7086940Z PLBartForConditionalGeneration PASS 2024-08-20T23:43:41.7092185Z PegasusForCausalLM PASS 2024-08-20T23:43:41.7097389Z PegasusForConditionalGeneration PASS 2024-08-20T23:43:41.7102504Z RobertaForCausalLM PASS 2024-08-20T23:43:41.7107906Z RobertaForQuestionAnswering PASS 2024-08-20T23:43:41.7112904Z Speech2Text2ForCausalLM PASS 2024-08-20T23:43:41.7119093Z T5ForConditionalGeneration PASS 2024-08-20T23:43:41.7124285Z T5Small PASS 2024-08-20T23:43:41.7129418Z TrOCRForCausalLM PASS 2024-08-20T23:43:41.7134878Z XGLMForCausalLM PASS 2024-08-20T23:43:41.7140085Z XLNetLMHeadModel PASS 2024-08-20T23:43:41.7145164Z YituTechConvBert PASS 2024-08-20T23:43:41.7573933Z + cleanup_workspace 2024-08-20T23:43:41.7575068Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:43:41.7576156Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:43:41.7577748Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:43:41.7578480Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:43:41.7579321Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:43:41.7580196Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:43:41.7580892Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:43:42.4893910Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:43:42.4894436Z cat test/**/*_toprint.log || true 2024-08-20T23:43:42.4908369Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:42.4908865Z env: 2024-08-20T23:43:42.4909139Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:42.4909594Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:42.4910314Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:42.4910948Z ##[endgroup] 2024-08-20T23:43:42.5006174Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:43:42.5038859Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:43:42.5039328Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:43:42.5048291Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:42.5048786Z env: 2024-08-20T23:43:42.5049055Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:42.5049508Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:42.5050360Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:42.5051021Z MONITOR_SCRIPT_PID: 52429 2024-08-20T23:43:42.5051364Z ##[endgroup] 2024-08-20T23:43:42.5245852Z Prepare all required actions 2024-08-20T23:43:42.5246296Z Getting action download info 2024-08-20T23:43:42.7273383Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:43:42.8851493Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:43:42.8851964Z with: 2024-08-20T23:43:42.8852554Z file-suffix: test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621 2024-08-20T23:43:42.8853265Z s3-bucket: gha-artifacts 2024-08-20T23:43:42.8853599Z env: 2024-08-20T23:43:42.8853874Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:42.8854325Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:42.8855055Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:42.8855679Z ##[endgroup] 2024-08-20T23:43:42.8881913Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:43:42.8882572Z # Remove any previous test jsons if they exist 2024-08-20T23:43:42.8883082Z rm -f test-jsons-*.zip 2024-08-20T23:43:42.8883597Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:43:42.8892681Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:42.8893169Z env: 2024-08-20T23:43:42.8893443Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:42.8893882Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:42.8894604Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:42.8895533Z FILE_SUFFIX: test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621 2024-08-20T23:43:42.8896220Z ##[endgroup] 2024-08-20T23:43:42.9256286Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:43:42.9285200Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:43:42.9286271Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:43:42.9292347Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:43:42.9297836Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:43:42.9816813Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:43:42.9817453Z # Remove any previous test reports if they exist 2024-08-20T23:43:42.9817980Z rm -f test-reports-*.zip 2024-08-20T23:43:42.9818569Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:43:42.9827883Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:42.9828425Z env: 2024-08-20T23:43:42.9828706Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:42.9829188Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:42.9829990Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:42.9831116Z FILE_SUFFIX: test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621 2024-08-20T23:43:42.9831880Z ##[endgroup] 2024-08-20T23:43:43.0096160Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-08-20T23:43:43.0097245Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%) 2024-08-20T23:43:43.0098208Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%) 2024-08-20T23:43:43.0099091Z adding: test/test-reports/training_huggingface.csv (deflated 74%) 2024-08-20T23:43:43.0105506Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-08-20T23:43:43.0106451Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%) 2024-08-20T23:43:43.0139502Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:43:43.0140135Z # Remove any previous usage logs if they exist 2024-08-20T23:43:43.0140633Z rm -f logs-*.zip 2024-08-20T23:43:43.0141528Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:43:43.0142316Z # so check to see if the file exists first 2024-08-20T23:43:43.0142812Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:43:43.0143433Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:43:43.0143934Z fi 2024-08-20T23:43:43.0144300Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:43:43.0144868Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:43:43.0145359Z fi 2024-08-20T23:43:43.0153879Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:43.0154373Z env: 2024-08-20T23:43:43.0154642Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:43.0155095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:43.0155822Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:43.0156768Z FILE_SUFFIX: test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621 2024-08-20T23:43:43.0157448Z ##[endgroup] 2024-08-20T23:43:43.0252955Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:43:43.0341168Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:43:43.0341880Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:43:43.0342428Z rm -f debug-*.zip 2024-08-20T23:43:43.0342811Z if [ -d 'test/debug' ]; then 2024-08-20T23:43:43.0343312Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:43:43.0343790Z fi 2024-08-20T23:43:43.0351993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:43.0352495Z env: 2024-08-20T23:43:43.0352767Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:43.0353222Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:43.0353954Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:43.0355083Z FILE_SUFFIX: test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621 2024-08-20T23:43:43.0355767Z ##[endgroup] 2024-08-20T23:43:43.0450412Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:43.0450848Z with: 2024-08-20T23:43:43.0451130Z s3-bucket: gha-artifacts 2024-08-20T23:43:43.0451563Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:43.0452029Z retention-days: 14 2024-08-20T23:43:43.0452358Z if-no-files-found: warn 2024-08-20T23:43:43.0452712Z path: test-jsons-*.zip 2024-08-20T23:43:43.0453050Z name: artifact 2024-08-20T23:43:43.0453342Z region: us-east-1 2024-08-20T23:43:43.0453641Z env: 2024-08-20T23:43:43.0453915Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:43.0454358Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:43.0455075Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:43.0455704Z ##[endgroup] 2024-08-20T23:43:43.4037732Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:43.4038378Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:43.4039002Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:43.5066473Z Starting upload of test-jsons-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:43.6155421Z Finished upload of test-jsons-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:43.6431066Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:43.6431515Z with: 2024-08-20T23:43:43.6431792Z s3-bucket: gha-artifacts 2024-08-20T23:43:43.6432232Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:43.6432707Z retention-days: 14 2024-08-20T23:43:43.6433047Z if-no-files-found: error 2024-08-20T23:43:43.6433409Z path: test-reports-*.zip 2024-08-20T23:43:43.6433761Z name: artifact 2024-08-20T23:43:43.6434082Z region: us-east-1 2024-08-20T23:43:43.6434386Z env: 2024-08-20T23:43:43.6434888Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:43.6435343Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:43.6436076Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:43.6436720Z ##[endgroup] 2024-08-20T23:43:43.9964524Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:43.9965233Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:43.9965918Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:44.0020247Z Starting upload of test-reports-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:44.0909824Z Finished upload of test-reports-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:44.1205504Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:44.1205945Z with: 2024-08-20T23:43:44.1206243Z s3-bucket: gha-artifacts 2024-08-20T23:43:44.1206691Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:44.1207166Z retention-days: 14 2024-08-20T23:43:44.1207508Z if-no-files-found: ignore 2024-08-20T23:43:44.1207861Z path: logs-*.zip 2024-08-20T23:43:44.1208172Z name: artifact 2024-08-20T23:43:44.1208474Z region: us-east-1 2024-08-20T23:43:44.1208770Z env: 2024-08-20T23:43:44.1209046Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:44.1209497Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:44.1210435Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:44.1211065Z ##[endgroup] 2024-08-20T23:43:44.4743700Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:44.4744291Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:44.4744910Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:44.4800338Z Starting upload of logs-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:44.6525677Z Finished upload of logs-test-inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527621.zip 2024-08-20T23:43:44.6800927Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:44.6801381Z with: 2024-08-20T23:43:44.6801665Z s3-bucket: gha-artifacts 2024-08-20T23:43:44.6802107Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:44.6802584Z retention-days: 14 2024-08-20T23:43:44.6802926Z if-no-files-found: ignore 2024-08-20T23:43:44.6803282Z path: debug-*.zip 2024-08-20T23:43:44.6803597Z name: artifact 2024-08-20T23:43:44.6803904Z region: us-east-1 2024-08-20T23:43:44.6804198Z env: 2024-08-20T23:43:44.6804481Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:44.6804939Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:44.6805673Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:44.6806321Z ##[endgroup] 2024-08-20T23:43:45.0289615Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:43:45.0566347Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:43:45.0566839Z # shellcheck disable=SC2156 2024-08-20T23:43:45.0567628Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:43:45.0578110Z shell: /usr/bin/bash -e {0} 2024-08-20T23:43:45.0578462Z env: 2024-08-20T23:43:45.0578744Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:45.0579202Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:45.0579926Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:45.0580566Z ##[endgroup] 2024-08-20T23:43:45.3201932Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:43:45.3202486Z with: 2024-08-20T23:43:45.3202746Z env: 2024-08-20T23:43:45.3203029Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:45.3203497Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:45.3204221Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:45.3204859Z ##[endgroup] 2024-08-20T23:43:45.3224329Z ##[group]Run set -eou pipefail 2024-08-20T23:43:45.3224733Z set -eou pipefail 2024-08-20T23:43:45.3225083Z  2024-08-20T23:43:45.3225586Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:43:45.3226212Z for _ in $(seq 1440); do 2024-08-20T23:43:45.3226683Z  # Break if no ssh session exists anymore 2024-08-20T23:43:45.3227182Z  if [ "$(who)" = "" ]; then 2024-08-20T23:43:45.3227589Z  break 2024-08-20T23:43:45.3227929Z  fi 2024-08-20T23:43:45.3228222Z  echo "." 2024-08-20T23:43:45.3228554Z  sleep 5 2024-08-20T23:43:45.3228867Z done 2024-08-20T23:43:45.3237500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:45.3238002Z env: 2024-08-20T23:43:45.3238281Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:45.3238734Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:45.3239460Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:45.3240086Z ##[endgroup] 2024-08-20T23:43:45.3268592Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:43:45.3368075Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:43:45.3368820Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:43:45.3369406Z # shellcheck disable=SC2046 2024-08-20T23:43:45.3370097Z docker stop $(docker ps -q) || true 2024-08-20T23:43:45.3370575Z # Prune all of the docker images 2024-08-20T23:43:45.3371040Z docker system prune -af 2024-08-20T23:43:45.3379175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:45.3379879Z env: 2024-08-20T23:43:45.3380156Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:45.3380634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:45.3381382Z DOCKER_CONTAINER_ID: cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:45.3382010Z ##[endgroup] 2024-08-20T23:43:45.9020637Z cdf17c7bd466 2024-08-20T23:43:46.6695096Z Deleted Containers: 2024-08-20T23:43:46.6695772Z cdf17c7bd4669de6742f306f837705980c19d991c6cd042edf0271581d9e4521 2024-08-20T23:43:46.6696305Z 2024-08-20T23:43:57.6656146Z Deleted Images: 2024-08-20T23:43:57.6658216Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:43:57.6660835Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:43:57.6662386Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:43:57.6663257Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:43:57.6664117Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:43:57.6664951Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:43:57.6665812Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:43:57.6666615Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:43:57.6667437Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:43:57.6668624Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:43:57.6669456Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:43:57.6670303Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:43:57.6671135Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:43:57.6671956Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:43:57.6672778Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:43:57.6673599Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:43:57.6674418Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:43:57.6675244Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:43:57.6676081Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:43:57.6677085Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:43:57.6677919Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:43:57.6678741Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:43:57.6679575Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:43:57.6680416Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:43:57.6681242Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:43:57.6682066Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:43:57.6682890Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:43:57.6683732Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:43:57.6684567Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:43:57.6685404Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:43:57.6686439Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:43:57.6687268Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:43:57.6688091Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:43:57.6688914Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:43:57.6689740Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:43:57.6690786Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:43:57.6691621Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:43:57.6692451Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:43:57.6693282Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:43:57.6694131Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:43:57.6695011Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:43:57.6695845Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:43:57.6696681Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:43:57.6697529Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:43:57.6698359Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:43:57.6699166Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:43:57.6700123Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:43:57.6700973Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:43:57.6701814Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:43:57.6702621Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:43:57.6703737Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:43:57.6704590Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:43:57.6705422Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:43:57.6706225Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:43:57.6707042Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:43:57.6707881Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:43:57.6708718Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:43:57.6709544Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:43:57.6710372Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:43:57.6711196Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:43:57.6712010Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:43:57.6712826Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:43:57.6713653Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:43:57.6714904Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:43:57.6715750Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:43:57.6716568Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:43:57.6717385Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:43:57.6718357Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:43:57.6719179Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:43:57.6720008Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:43:57.6720839Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:43:57.6721664Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:43:57.6722469Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:43:57.6723287Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:43:57.6724139Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:43:57.6724987Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:43:57.6725831Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:43:57.6726670Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:43:57.6727508Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:43:57.6728349Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:43:57.6729173Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:43:57.6729664Z 2024-08-20T23:43:57.6729926Z Total reclaimed space: 39.92GB 2024-08-20T23:43:57.6798379Z Post job cleanup. 2024-08-20T23:43:57.6849152Z Post job cleanup. 2024-08-20T23:43:57.7732483Z [command]/usr/bin/git version 2024-08-20T23:43:57.7783390Z git version 2.40.1 2024-08-20T23:43:57.7821493Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/69df3427-e46e-4a2b-8c69-bc2a361e6050' before making global git config changes 2024-08-20T23:43:57.7822758Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:43:57.7826455Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:43:57.7873991Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:43:57.7922489Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:43:57.8290841Z Entering 'android/libs/fbjni' 2024-08-20T23:43:57.8360448Z Entering 'third_party/FP16' 2024-08-20T23:43:57.8429851Z Entering 'third_party/FXdiv' 2024-08-20T23:43:57.8499248Z Entering 'third_party/NNPACK' 2024-08-20T23:43:57.8568543Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:43:57.8637085Z Entering 'third_party/XNNPACK' 2024-08-20T23:43:57.8722368Z Entering 'third_party/benchmark' 2024-08-20T23:43:57.8795886Z Entering 'third_party/cpp-httplib' 2024-08-20T23:43:57.8869341Z Entering 'third_party/cpuinfo' 2024-08-20T23:43:57.8937744Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:43:57.9006157Z Entering 'third_party/cutlass' 2024-08-20T23:43:57.9081576Z Entering 'third_party/eigen' 2024-08-20T23:43:57.9151503Z Entering 'third_party/fbgemm' 2024-08-20T23:43:57.9219535Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:43:57.9284619Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:43:57.9351548Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:43:57.9425242Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:43:57.9491045Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:43:57.9561356Z Entering 'third_party/flatbuffers' 2024-08-20T23:43:57.9633720Z Entering 'third_party/fmt' 2024-08-20T23:43:57.9701614Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:43:57.9772732Z Entering 'third_party/gloo' 2024-08-20T23:43:57.9842073Z Entering 'third_party/googletest' 2024-08-20T23:43:57.9911254Z Entering 'third_party/ideep' 2024-08-20T23:43:57.9979923Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:43:58.0055394Z Entering 'third_party/ittapi' 2024-08-20T23:43:58.0125240Z Entering 'third_party/kineto' 2024-08-20T23:43:58.0194074Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:43:58.0264685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:43:58.0336005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:43:58.0404356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:43:58.0471782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:43:58.0537334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:43:58.0609107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:43:58.0678436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:43:58.0747698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:43:58.0817803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:43:58.0887684Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:43:58.0954700Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:43:58.1024184Z Entering 'third_party/mimalloc' 2024-08-20T23:43:58.1094045Z Entering 'third_party/nccl/nccl' 2024-08-20T23:43:58.1163190Z Entering 'third_party/nlohmann' 2024-08-20T23:43:58.1235089Z Entering 'third_party/onnx' 2024-08-20T23:43:58.1316921Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:43:58.1383454Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:43:58.1462731Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:43:58.1531007Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:43:58.1598567Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:43:58.1666058Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:43:58.1735340Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:43:58.1803873Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:43:58.1871379Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:43:58.1939742Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:43:58.2004598Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:43:58.2073972Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:43:58.2148313Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:43:58.2237817Z Entering 'third_party/pocketfft' 2024-08-20T23:43:58.2307438Z Entering 'third_party/protobuf' 2024-08-20T23:43:58.2378936Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:43:58.2445768Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:43:58.2519194Z Entering 'third_party/psimd' 2024-08-20T23:43:58.2589577Z Entering 'third_party/pthreadpool' 2024-08-20T23:43:58.2659340Z Entering 'third_party/pybind11' 2024-08-20T23:43:58.2729022Z Entering 'third_party/python-peachpy' 2024-08-20T23:43:58.2799176Z Entering 'third_party/sleef' 2024-08-20T23:43:58.2868487Z Entering 'third_party/tensorpipe' 2024-08-20T23:43:58.2936798Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:43:58.3004087Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:43:58.3069457Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:43:58.3139471Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:43:58.3202430Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:43:58.3298568Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:43:58.3336028Z http.https://github.com/.extraheader 2024-08-20T23:43:58.3346864Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:43:58.3395145Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:43:58.3761291Z Entering 'android/libs/fbjni' 2024-08-20T23:43:58.3804357Z http.https://github.com/.extraheader 2024-08-20T23:43:58.3848609Z Entering 'third_party/FP16' 2024-08-20T23:43:58.3893156Z http.https://github.com/.extraheader 2024-08-20T23:43:58.3940274Z Entering 'third_party/FXdiv' 2024-08-20T23:43:58.3985041Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4028311Z Entering 'third_party/NNPACK' 2024-08-20T23:43:58.4071982Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4114880Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:43:58.4161409Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4205043Z Entering 'third_party/XNNPACK' 2024-08-20T23:43:58.4248702Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4307789Z Entering 'third_party/benchmark' 2024-08-20T23:43:58.4351878Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4395653Z Entering 'third_party/cpp-httplib' 2024-08-20T23:43:58.4439369Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4484478Z Entering 'third_party/cpuinfo' 2024-08-20T23:43:58.4529094Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4574438Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:43:58.4618302Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4661460Z Entering 'third_party/cutlass' 2024-08-20T23:43:58.4704012Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4755402Z Entering 'third_party/eigen' 2024-08-20T23:43:58.4798845Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4846421Z Entering 'third_party/fbgemm' 2024-08-20T23:43:58.4894977Z http.https://github.com/.extraheader 2024-08-20T23:43:58.4939948Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:43:58.4982963Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5027018Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:43:58.5069700Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5112772Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:43:58.5155807Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5205200Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:43:58.5248459Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5290798Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:43:58.5334366Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5379834Z Entering 'third_party/flatbuffers' 2024-08-20T23:43:58.5425577Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5472242Z Entering 'third_party/fmt' 2024-08-20T23:43:58.5517407Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5561254Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:43:58.5605689Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5661885Z Entering 'third_party/gloo' 2024-08-20T23:43:58.5704881Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5750084Z Entering 'third_party/googletest' 2024-08-20T23:43:58.5795449Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5850222Z Entering 'third_party/ideep' 2024-08-20T23:43:58.5894826Z http.https://github.com/.extraheader 2024-08-20T23:43:58.5938701Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:43:58.5983033Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6034989Z Entering 'third_party/ittapi' 2024-08-20T23:43:58.6079235Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6123529Z Entering 'third_party/kineto' 2024-08-20T23:43:58.6167371Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6208941Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:43:58.6253167Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6296309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:43:58.6340177Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6385843Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:43:58.6430497Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6475192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:43:58.6518929Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6564878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:43:58.6609052Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6652802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:43:58.6696335Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6744629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:43:58.6789516Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6833888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:43:58.6876317Z http.https://github.com/.extraheader 2024-08-20T23:43:58.6921119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:43:58.6963076Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7007123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:43:58.7052293Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7099772Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:43:58.7142829Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7186063Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:43:58.7228218Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7274302Z Entering 'third_party/mimalloc' 2024-08-20T23:43:58.7320172Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7364519Z Entering 'third_party/nccl/nccl' 2024-08-20T23:43:58.7408451Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7453239Z Entering 'third_party/nlohmann' 2024-08-20T23:43:58.7497680Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7544851Z Entering 'third_party/onnx' 2024-08-20T23:43:58.7589781Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7646901Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:43:58.7690263Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7734749Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:43:58.7778749Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7830946Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:43:58.7874867Z http.https://github.com/.extraheader 2024-08-20T23:43:58.7920171Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:43:58.7961669Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8004536Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:43:58.8048745Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8092056Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:43:58.8134676Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8177808Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:43:58.8221178Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8265080Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:43:58.8308308Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8351866Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:43:58.8393176Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8436765Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:43:58.8477558Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8519157Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:43:58.8562307Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8607782Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:43:58.8653922Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8701466Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:43:58.8743482Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8810499Z Entering 'third_party/pocketfft' 2024-08-20T23:43:58.8854822Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8898933Z Entering 'third_party/protobuf' 2024-08-20T23:43:58.8945719Z http.https://github.com/.extraheader 2024-08-20T23:43:58.8990463Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:43:58.9033741Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9077213Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:43:58.9119464Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9166608Z Entering 'third_party/psimd' 2024-08-20T23:43:58.9214586Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9259960Z Entering 'third_party/pthreadpool' 2024-08-20T23:43:58.9303512Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9347278Z Entering 'third_party/pybind11' 2024-08-20T23:43:58.9390439Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9436441Z Entering 'third_party/python-peachpy' 2024-08-20T23:43:58.9479807Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9524452Z Entering 'third_party/sleef' 2024-08-20T23:43:58.9567414Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9610254Z Entering 'third_party/tensorpipe' 2024-08-20T23:43:58.9655057Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9697250Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:43:58.9742234Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9786113Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:43:58.9828825Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9872221Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:43:58.9915512Z http.https://github.com/.extraheader 2024-08-20T23:43:58.9959610Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:43:59.0001709Z http.https://github.com/.extraheader 2024-08-20T23:43:59.0044038Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:43:59.0087513Z http.https://github.com/.extraheader 2024-08-20T23:43:59.0246395Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:43:59.0266640Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:43:59.0274689Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:59.0275207Z ##[endgroup] 2024-08-20T23:44:05.9227681Z Cleaning up orphan processes