2024-08-20T22:10:08.7555049Z Current runner version: '2.319.1' 2024-08-20T22:10:08.7561195Z Runner name: 'i-024c42853ba21bbc0' 2024-08-20T22:10:08.7562014Z Runner group name: 'Default' 2024-08-20T22:10:08.7562936Z Machine name: 'ip-10-0-56-249' 2024-08-20T22:10:08.7579884Z Testing runner upgrade compatibility 2024-08-20T22:10:08.9214688Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:10:08.9216755Z Actions: read 2024-08-20T22:10:08.9217241Z Attestations: read 2024-08-20T22:10:08.9217794Z Checks: read 2024-08-20T22:10:08.9218250Z Contents: read 2024-08-20T22:10:08.9218719Z Deployments: read 2024-08-20T22:10:08.9219239Z Discussions: read 2024-08-20T22:10:08.9219704Z Issues: read 2024-08-20T22:10:08.9220144Z Metadata: read 2024-08-20T22:10:08.9220678Z Packages: read 2024-08-20T22:10:08.9221096Z Pages: read 2024-08-20T22:10:08.9221541Z PullRequests: read 2024-08-20T22:10:08.9222121Z RepositoryProjects: read 2024-08-20T22:10:08.9222626Z SecurityEvents: read 2024-08-20T22:10:08.9223137Z Statuses: read 2024-08-20T22:10:08.9223833Z ##[endgroup] 2024-08-20T22:10:08.9227085Z Secret source: Actions 2024-08-20T22:10:08.9227740Z Prepare workflow directory 2024-08-20T22:10:09.0150909Z Prepare all required actions 2024-08-20T22:10:09.0315711Z Getting action download info 2024-08-20T22:10:09.1856040Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:10:09.6760592Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:13.6467412Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:13.8355328Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:14.1819008Z Getting action download info 2024-08-20T22:10:14.2783046Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:14.5481276Z Getting action download info 2024-08-20T22:10:14.6371825Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:14.8142667Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:14.8144986Z ##[group] Inputs 2024-08-20T22:10:14.8145510Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:14.8154994Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:14.8165301Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:14.8166485Z sync-tag: 2024-08-20T22:10:14.8167293Z timeout-minutes: 240 2024-08-20T22:10:14.8167629Z use-gha: 2024-08-20T22:10:14.8167931Z dashboard-tag: 2024-08-20T22:10:14.8168264Z s3-bucket: gha-artifacts 2024-08-20T22:10:14.8168620Z aws-role-to-assume: 2024-08-20T22:10:14.8168958Z ##[endgroup] 2024-08-20T22:10:14.8169887Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:14.8883606Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:14.9023494Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:14.9035207Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:14.9035748Z ##[endgroup] 2024-08-20T22:10:16.1037177Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:16.1037739Z Instance Type: g5.4xlarge 2024-08-20T22:10:16.1038410Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:16.1038943Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:21.8998299Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:21.8998844Z with: 2024-08-20T22:10:21.8999540Z github-secret: *** 2024-08-20T22:10:21.9000460Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:21.9001442Z activate-with-label: false 2024-08-20T22:10:21.9001789Z label: with-ssh 2024-08-20T22:10:21.9002106Z remove-existing-keys: true 2024-08-20T22:10:21.9002469Z fail-silently: true 2024-08-20T22:10:21.9002796Z env: 2024-08-20T22:10:21.9003091Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:21.9003427Z ##[endgroup] 2024-08-20T22:10:21.9897767Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:21.9900951Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:22.2368338Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:22.3028495Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:22.3029118Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:22.3677990Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:22.3692541Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:22.3717288Z Login using: ssh ec2-user@ec2-54-160-237-153.compute-1.amazonaws.com 2024-08-20T22:10:22.3718536Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:22.3719582Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:22.3848623Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:22.3849174Z with: 2024-08-20T22:10:22.3849451Z submodules: recursive 2024-08-20T22:10:22.3849779Z fetch-depth: 0 2024-08-20T22:10:22.3850065Z env: 2024-08-20T22:10:22.3850336Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:22.3850668Z ##[endgroup] 2024-08-20T22:10:22.4040379Z ##[group]Run retry () { 2024-08-20T22:10:22.4040762Z retry () { 2024-08-20T22:10:22.4041459Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:22.4042008Z } 2024-08-20T22:10:22.4042309Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:22.4042724Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:22.4043199Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:22.4043646Z else 2024-08-20T22:10:22.4043983Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:22.4044403Z fi 2024-08-20T22:10:22.4044735Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:22.4057406Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:22.4057905Z env: 2024-08-20T22:10:22.4058177Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:22.4058511Z NO_SUDO: 2024-08-20T22:10:22.4058782Z ##[endgroup] 2024-08-20T22:10:22.4091081Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:22.5393597Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:22.5394039Z with: 2024-08-20T22:10:22.5394362Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:22.5394799Z fetch-depth: 0 2024-08-20T22:10:22.5395223Z submodules: recursive 2024-08-20T22:10:22.5395555Z quiet-checkout: true 2024-08-20T22:10:22.5395903Z repository: pytorch/pytorch 2024-08-20T22:10:22.5396372Z token: *** 2024-08-20T22:10:22.5396659Z ssh-strict: true 2024-08-20T22:10:22.5396973Z persist-credentials: true 2024-08-20T22:10:22.5397322Z clean: true 2024-08-20T22:10:22.5397648Z sparse-checkout-cone-mode: true 2024-08-20T22:10:22.5398030Z lfs: false 2024-08-20T22:10:22.5398319Z set-safe-directory: true 2024-08-20T22:10:22.5398652Z env: 2024-08-20T22:10:22.5398918Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:22.5399249Z ##[endgroup] 2024-08-20T22:10:22.6378158Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:22.6379825Z ##[group]Getting Git version info 2024-08-20T22:10:22.6380640Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:22.6381660Z [command]/usr/bin/git version 2024-08-20T22:10:22.6382020Z git version 2.40.1 2024-08-20T22:10:22.6383580Z ##[endgroup] 2024-08-20T22:10:22.6395365Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/83586df6-27f6-4634-8e13-ad4f9d219582' before making global git config changes 2024-08-20T22:10:22.6396595Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:22.6398754Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:22.6448198Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:22.6452125Z ##[group]Initializing the repository 2024-08-20T22:10:22.6454967Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:22.6498008Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:22.6498978Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:22.6499783Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:22.6500348Z hint: 2024-08-20T22:10:22.6500796Z hint: git config --global init.defaultBranch 2024-08-20T22:10:22.6501269Z hint: 2024-08-20T22:10:22.6501794Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:22.6502778Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:22.6503808Z hint: 2024-08-20T22:10:22.6504180Z hint: git branch -m 2024-08-20T22:10:22.6505064Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:22.6511346Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:22.6552463Z ##[endgroup] 2024-08-20T22:10:22.6553077Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:22.6555253Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:22.6596089Z ##[endgroup] 2024-08-20T22:10:22.6596612Z ##[group]Setting up auth 2024-08-20T22:10:22.6601823Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:22.6643013Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:22.7027479Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:22.7067349Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:22.7420399Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:22.7490968Z ##[endgroup] 2024-08-20T22:10:22.7491538Z ##[group]Fetching the repository 2024-08-20T22:10:22.7497226Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:25.7750643Z remote: Enumerating objects: 1008509 2024-08-20T22:10:25.7751341Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:25.7753345Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:25.7754081Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:25.7754779Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:25.7755477Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:25.7756346Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:25.7757000Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:25.7757518Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:25.7758022Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:25.7758718Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:25.7759374Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:25.7760026Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:25.7760637Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:25.7761309Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:25.7761969Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:25.7762546Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:25.7763161Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:25.7763830Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:25.7764455Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:25.7765118Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:25.7765772Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:25.7766437Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:25.7767102Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:25.7767758Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:25.7768251Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:25.7768751Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:25.7769406Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:25.7770059Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:25.7770701Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:25.7771594Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:25.7772475Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:25.7773003Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:25.7773487Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:25.7774142Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:25.7774769Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:25.7775553Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:25.7776198Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:25.7776761Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:25.7777250Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:25.7777741Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:25.7778224Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:25.7778711Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:25.7779205Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:25.7779697Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:25.7780180Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:25.7780691Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:25.7781177Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:25.7781662Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:25.7782148Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:25.7782637Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:25.7783123Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:25.7783620Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:25.7784207Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:25.7784698Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:25.7785189Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:25.7785679Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:25.7786159Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:25.7786647Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:25.7787139Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:25.7787622Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:25.7788111Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:25.7788635Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:25.7789148Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:25.7789659Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:25.7790177Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:25.7790678Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:25.7791186Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:25.7791694Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:25.7792198Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:25.7792699Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:25.7793199Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:25.7793699Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:25.7794209Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:25.7794756Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:25.7795262Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:25.7795764Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:25.7796268Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:25.7796763Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:25.7797265Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:25.7797766Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:25.7798420Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:25.7798920Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:25.7799425Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:25.7799926Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:25.7800427Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:25.7800922Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:25.7801423Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:25.7802046Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:25.7802549Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:25.7803047Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:25.7803547Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:25.7804049Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:25.7804552Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:25.7805054Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:25.7805554Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:25.7806054Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:25.7806551Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:25.7807045Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:25.7807546Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:25.7808052Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:25.7808546Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:25.7809047Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:25.7809584Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:25.7924636Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:25.8185658Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:25.8472862Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:25.8930882Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:25.9743446Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:26.0247058Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:26.0951068Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:26.1283237Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:26.1669886Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:26.1881315Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:26.2128939Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:26.2362132Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:26.2540378Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:26.2703203Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:26.2826960Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:26.2908889Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:26.3000101Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:26.3072790Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:26.3116044Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:26.3164756Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:26.3197102Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:26.3217640Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:26.3222444Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:26.3228451Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:26.3237191Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:26.3242744Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:26.3253749Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:26.3258658Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:26.3279477Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:26.3292932Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:26.3302932Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:26.3314195Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:26.3318623Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:26.3329919Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:26.3337512Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:26.3341549Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:26.3350006Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:26.3356920Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:26.3366632Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:26.3373874Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:26.3379499Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:26.3385782Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:26.3394694Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:26.3398816Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:26.3407038Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:26.3411628Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:26.3416892Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:26.3421129Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:26.3423967Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:26.3428283Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:26.3431020Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:26.3434447Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:26.3437098Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:26.3438972Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:26.3440356Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:26.3441839Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:26.3443160Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:26.3443973Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:26.3445361Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:26.3446372Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:26.3447261Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:26.3447972Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:26.3448664Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:26.3449352Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:26.3451797Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:26.3458383Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:26.3465284Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:26.3470680Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:26.3474618Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:26.3478287Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:26.3482367Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:26.3486711Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:26.3490948Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:26.3494698Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:26.3495981Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:26.3499886Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:26.3504259Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:26.3507367Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:26.3509496Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:26.3511326Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:26.3512520Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:26.3515559Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:26.3517485Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:26.3519583Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:26.3521634Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:26.3524937Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:26.3526390Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:26.3528543Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:26.3530459Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:26.3533519Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:26.3534735Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:26.3536269Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:26.3536967Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:26.3537664Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:26.3538349Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:26.3539122Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:26.3540893Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:26.3541985Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:26.3543221Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:26.3544580Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:26.3545798Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:26.3546533Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:49.6735648Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:11:13.0992634Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:13.1029784Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:13.1035098Z ##[endgroup] 2024-08-20T22:11:13.1037790Z ##[group]Determining the checkout info 2024-08-20T22:11:13.1038444Z ##[endgroup] 2024-08-20T22:11:13.1038951Z ##[group]Checking out the ref 2024-08-20T22:11:13.1040112Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:14.7765117Z ##[endgroup] 2024-08-20T22:11:14.7765715Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:14.7773508Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:14.7838633Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:14.7877311Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:14.7915692Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:14.7951813Z ##[endgroup] 2024-08-20T22:11:14.7952348Z ##[group]Fetching submodules 2024-08-20T22:11:14.7955384Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:14.8332346Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:14.8698277Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:14.8699818Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:14.8703383Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:14.8707430Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:14.8711848Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:14.8715016Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:14.8718814Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:14.8723915Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:14.8728164Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:14.8732094Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:14.8736208Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:14.8740245Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:14.8744533Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:14.8748987Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:14.8753416Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:14.8757910Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:14.8765920Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:14.8770448Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:14.8775675Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:14.8780410Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:14.8785323Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:14.8790528Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:14.8795137Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:14.8800457Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:14.8805161Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:14.8810673Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:14.8815649Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:14.8820953Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:14.8826479Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:14.8831951Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:14.8837331Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:14.8843090Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:14.8848420Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:14.8857514Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:14.8892782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:15.1914742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:15.3703425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:15.5640990Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:15.8092707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:17.6835732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:28.9716095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:29.3938760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:29.8167593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:30.4382879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:31.6747273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:33.7865749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:39.6826367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:41.1204850Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:42.8521486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:44.1499033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:44.5653522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:44.8829275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:46.0010650Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:46.3839883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:46.6421483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:48.3401076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:49.1364367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:49.8316248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:55.9120342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:57.9506646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:12:02.2072973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:12:02.4526533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:11.2717323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:11.4620117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:11.6559516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:12.5764031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:12.8851956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:13.5154515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:14.0970818Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:14.1120190Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:14.1238227Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:14.1550337Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:14.2006679Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:15.3727777Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:15.4011639Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:15.4543370Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:15.5669958Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:15.6071521Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:16.2295452Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:16.5159259Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:16.6154181Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:16.6180527Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:16.6183945Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:16.6188016Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:16.6191650Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:16.6195328Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:16.6227728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:17.6583883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:18.2627605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:20.3561867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:21.4005165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:21.7390189Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:21.8472425Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:22.3523410Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:22.4226771Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:22.4376386Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:22.5883550Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:22.6346896Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:22.6794833Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:22.7111956Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:22.7629964Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:22.7787586Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:22.7807638Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:22.7842449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:35.8559762Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:35.8777534Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:35.9772988Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:35.9795520Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:35.9798651Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:35.9802234Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:35.9834653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:36.5637854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:37.7436108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:38.9000158Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:38.9020050Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:38.9023327Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:38.9027193Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:38.9030975Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:38.9034666Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:38.9038479Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:38.9042259Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:38.9046149Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:38.9078886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:39.7887317Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:40.1657895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:41.3588398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:41.6527753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:42.1934193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:43.2759853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:50.8851715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:51.3538452Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:51.3774159Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:51.4216708Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:51.4384120Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:51.4404394Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:51.4435599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:51.7739684Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:51.7966094Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:51.8446283Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:51.9627067Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:51.9828653Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:52.0282216Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:52.0944419Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:52.1390234Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:52.1686275Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:52.2933091Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:52.8036020Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:52.8077292Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:52.8078840Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:52.8110494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:53.2820484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:54.2374924Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:54.2781380Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:54.3673364Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:54.3695988Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:54.3699817Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:54.3703154Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:54.3706936Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:54.3710591Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:54.3714321Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:54.3717840Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:54.3721393Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:54.3753127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:54.8294638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:55.8486748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:56.1656303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:13:02.2329150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:13:02.4998741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:13:02.7313656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:13:03.0445499Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:13:09.9366296Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:13:09.9839509Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:13:10.0029064Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:13:10.1255196Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:13:10.1418447Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:13:10.1601710Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:13:10.1805457Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:13:10.1824844Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:10.1828257Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:10.1858648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:12.1031245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:13.4607726Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:13.5126620Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:14.1619473Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:14.1762301Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:14.4903738Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:14.4930022Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:14.4933475Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:14.4965344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:14.9271044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:15.9824586Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:16.0637334Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:16.0756815Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:16.0910729Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:16.1352815Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:16.1697205Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:16.2185291Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:16.2527122Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:16.2548633Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:16.2551945Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:16.2555572Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:16.2558895Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:16.2589937Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:17.3858016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:17.6330375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:18.8083120Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:19.8029994Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:19.8224316Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:19.9017100Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:19.9364546Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:19.9385088Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:19.9416186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:20.1711349Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:20.1755675Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:20.2124650Z Entering 'android/libs/fbjni' 2024-08-20T22:13:20.2176760Z Entering 'third_party/FP16' 2024-08-20T22:13:20.2227912Z Entering 'third_party/FXdiv' 2024-08-20T22:13:20.2279623Z Entering 'third_party/NNPACK' 2024-08-20T22:13:20.2330668Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:20.2382742Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:20.2451266Z Entering 'third_party/benchmark' 2024-08-20T22:13:20.2504858Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:20.2556570Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:20.2607945Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:20.2658752Z Entering 'third_party/cutlass' 2024-08-20T22:13:20.2721778Z Entering 'third_party/eigen' 2024-08-20T22:13:20.2775369Z Entering 'third_party/fbgemm' 2024-08-20T22:13:20.2825366Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:20.2876166Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:20.2925097Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:20.2981014Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:20.3029656Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:20.3081192Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:20.3134848Z Entering 'third_party/fmt' 2024-08-20T22:13:20.3191489Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:20.3242002Z Entering 'third_party/gloo' 2024-08-20T22:13:20.3294874Z Entering 'third_party/googletest' 2024-08-20T22:13:20.3345848Z Entering 'third_party/ideep' 2024-08-20T22:13:20.3394646Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:20.3453216Z Entering 'third_party/ittapi' 2024-08-20T22:13:20.3504062Z Entering 'third_party/kineto' 2024-08-20T22:13:20.3554271Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:20.3604422Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:20.3655394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:20.3711179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:20.3761170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:20.3810874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:20.3865700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:20.3916762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:20.3967155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:20.4019520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:20.4075883Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:20.4124267Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:20.4176285Z Entering 'third_party/mimalloc' 2024-08-20T22:13:20.4231887Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:20.4283562Z Entering 'third_party/nlohmann' 2024-08-20T22:13:20.4338206Z Entering 'third_party/onnx' 2024-08-20T22:13:20.4403030Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:20.4454857Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:20.4512226Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:20.4563617Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:20.4614894Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:20.4664123Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:20.4714217Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:20.4764695Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:20.4814521Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:20.4863503Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:20.4911559Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:20.4964106Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:20.5017906Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:20.5089055Z Entering 'third_party/pocketfft' 2024-08-20T22:13:20.5140154Z Entering 'third_party/protobuf' 2024-08-20T22:13:20.5193401Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:20.5244733Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:20.5299013Z Entering 'third_party/psimd' 2024-08-20T22:13:20.5354952Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:20.5406770Z Entering 'third_party/pybind11' 2024-08-20T22:13:20.5462641Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:20.5519362Z Entering 'third_party/sleef' 2024-08-20T22:13:20.5570369Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:20.5619943Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:20.5670485Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:20.5720461Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:20.5770531Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:20.5819050Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:20.5892222Z ##[endgroup] 2024-08-20T22:13:20.5898506Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:20.5900255Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:20.6255075Z Entering 'android/libs/fbjni' 2024-08-20T22:13:20.6324055Z Entering 'third_party/FP16' 2024-08-20T22:13:20.6392142Z Entering 'third_party/FXdiv' 2024-08-20T22:13:20.6459504Z Entering 'third_party/NNPACK' 2024-08-20T22:13:20.6528380Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:20.6595176Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:20.6677763Z Entering 'third_party/benchmark' 2024-08-20T22:13:20.6744831Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:20.6813566Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:20.6881326Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:20.6948718Z Entering 'third_party/cutlass' 2024-08-20T22:13:20.7024363Z Entering 'third_party/eigen' 2024-08-20T22:13:20.7093565Z Entering 'third_party/fbgemm' 2024-08-20T22:13:20.7159891Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:20.7225097Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:20.7291159Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:20.7361457Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:20.7427284Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:20.7495970Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:20.7566225Z Entering 'third_party/fmt' 2024-08-20T22:13:20.7634949Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:20.7702443Z Entering 'third_party/gloo' 2024-08-20T22:13:20.7771562Z Entering 'third_party/googletest' 2024-08-20T22:13:20.7840698Z Entering 'third_party/ideep' 2024-08-20T22:13:20.7906340Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:20.7981785Z Entering 'third_party/ittapi' 2024-08-20T22:13:20.8049086Z Entering 'third_party/kineto' 2024-08-20T22:13:20.8115351Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:20.8179499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:20.8249400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:20.8316869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:20.8386502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:20.8450782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:20.8521229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:20.8589916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:20.8656220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:20.8727456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:20.8797530Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:20.8867319Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:20.8937573Z Entering 'third_party/mimalloc' 2024-08-20T22:13:20.9005574Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:20.9074639Z Entering 'third_party/nlohmann' 2024-08-20T22:13:20.9142932Z Entering 'third_party/onnx' 2024-08-20T22:13:20.9223704Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:20.9290966Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:20.9363414Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:20.9432062Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:20.9499077Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:20.9567467Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:20.9635596Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:20.9702669Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:20.9772841Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:20.9837359Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:20.9906138Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:20.9975536Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:21.0045482Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:21.0138096Z Entering 'third_party/pocketfft' 2024-08-20T22:13:21.0208156Z Entering 'third_party/protobuf' 2024-08-20T22:13:21.0277824Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:21.0342836Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:21.0413858Z Entering 'third_party/psimd' 2024-08-20T22:13:21.0482454Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:21.0550704Z Entering 'third_party/pybind11' 2024-08-20T22:13:21.0618842Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:21.0688375Z Entering 'third_party/sleef' 2024-08-20T22:13:21.0755773Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:21.0823163Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:21.0888437Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:21.0954349Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:21.1020833Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:21.1089882Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:21.1182014Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:21.1548267Z Entering 'android/libs/fbjni' 2024-08-20T22:13:21.1609441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:21.1631040Z Entering 'third_party/FP16' 2024-08-20T22:13:21.1694086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:21.1715338Z Entering 'third_party/FXdiv' 2024-08-20T22:13:21.1776353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:21.1797638Z Entering 'third_party/NNPACK' 2024-08-20T22:13:21.1863416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:21.1885947Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:21.1948401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:21.1969675Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:21.2030662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:21.2067563Z Entering 'third_party/benchmark' 2024-08-20T22:13:21.2129677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:21.2150394Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:21.2212597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:21.2233661Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:21.2296048Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:21.2318310Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:21.2379105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:21.2400488Z Entering 'third_party/cutlass' 2024-08-20T22:13:21.2461396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:21.2490207Z Entering 'third_party/eigen' 2024-08-20T22:13:21.2550782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:21.2574776Z Entering 'third_party/fbgemm' 2024-08-20T22:13:21.2637285Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:21.2656823Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:21.2722904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:21.2742857Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:21.2805805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:21.2826001Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:21.2886366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:21.2912600Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:21.2973572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.2993797Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:21.3053486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:21.3077403Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:21.3140582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:21.3162744Z Entering 'third_party/fmt' 2024-08-20T22:13:21.3223810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:21.3245106Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:21.3306476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:21.3327790Z Entering 'third_party/gloo' 2024-08-20T22:13:21.3389152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:21.3410547Z Entering 'third_party/googletest' 2024-08-20T22:13:21.3471348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.3493104Z Entering 'third_party/ideep' 2024-08-20T22:13:21.3554216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:21.3573206Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:21.3634378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:21.3663721Z Entering 'third_party/ittapi' 2024-08-20T22:13:21.3726291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:21.3747242Z Entering 'third_party/kineto' 2024-08-20T22:13:21.3809369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:21.3829094Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:21.3896581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:21.3915133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:21.3978085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:21.3999242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:21.4060783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:21.4081844Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:21.4143405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:21.4163535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:21.4226720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:21.4247029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:21.4310706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:21.4334458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:21.4396810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:21.4416939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:21.4478522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.4500045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:21.4564152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:21.4586027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:21.4649818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:21.4675551Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:21.4736830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:21.4758383Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:21.4820195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.4844066Z Entering 'third_party/mimalloc' 2024-08-20T22:13:21.4906435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:21.4929474Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:21.4993075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:21.5014753Z Entering 'third_party/nlohmann' 2024-08-20T22:13:21.5077357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:21.5100186Z Entering 'third_party/onnx' 2024-08-20T22:13:21.5162879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:21.5198446Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:21.5264008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:21.5286883Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:21.5350629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:21.5375585Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:21.5438528Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:21.5459818Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:21.5520461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:21.5540267Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:21.5601592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.5620969Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:21.5686489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:21.5706267Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:21.5765766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:21.5789267Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:21.5849709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:21.5869130Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:21.5930814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:21.5950491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:21.6012465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:21.6029915Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:21.6091516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:21.6113460Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:21.6177708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:21.6200463Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:21.6261133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:21.6301208Z Entering 'third_party/pocketfft' 2024-08-20T22:13:21.6366475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:21.6388377Z Entering 'third_party/protobuf' 2024-08-20T22:13:21.6450850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:21.6474209Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:21.6534842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:21.6559666Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:21.6622122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.6644942Z Entering 'third_party/psimd' 2024-08-20T22:13:21.6709041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:21.6731575Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:21.6794182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:21.6815403Z Entering 'third_party/pybind11' 2024-08-20T22:13:21.6877886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:21.6899249Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:21.6960588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:21.6982478Z Entering 'third_party/sleef' 2024-08-20T22:13:21.7043929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:21.7065532Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:21.7128167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:21.7148414Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:21.7211384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:21.7231392Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:21.7292635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:21.7312101Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:21.7373143Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:21.7393363Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:21.7453848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:21.7471324Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:21.7534246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:21.8657125Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:21.9027209Z Entering 'android/libs/fbjni' 2024-08-20T22:13:21.9077974Z Entering 'third_party/FP16' 2024-08-20T22:13:21.9129926Z Entering 'third_party/FXdiv' 2024-08-20T22:13:21.9182586Z Entering 'third_party/NNPACK' 2024-08-20T22:13:21.9234720Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:21.9286821Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:21.9355442Z Entering 'third_party/benchmark' 2024-08-20T22:13:21.9407240Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:21.9458626Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:21.9511457Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:21.9562979Z Entering 'third_party/cutlass' 2024-08-20T22:13:21.9622815Z Entering 'third_party/eigen' 2024-08-20T22:13:21.9677521Z Entering 'third_party/fbgemm' 2024-08-20T22:13:21.9730556Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:21.9781874Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:21.9832920Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:21.9889323Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:21.9939460Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:21.9993464Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:22.0047927Z Entering 'third_party/fmt' 2024-08-20T22:13:22.0101205Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:22.0152538Z Entering 'third_party/gloo' 2024-08-20T22:13:22.0205147Z Entering 'third_party/googletest' 2024-08-20T22:13:22.0256560Z Entering 'third_party/ideep' 2024-08-20T22:13:22.0307279Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:22.0365909Z Entering 'third_party/ittapi' 2024-08-20T22:13:22.0418668Z Entering 'third_party/kineto' 2024-08-20T22:13:22.0469672Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:22.0518789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:22.0570892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:22.0621737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:22.0672502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:22.0723072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:22.0777256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:22.0829322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:22.0880317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:22.0932040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:22.0987122Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:22.1040706Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:22.1094015Z Entering 'third_party/mimalloc' 2024-08-20T22:13:22.1146794Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:22.1199942Z Entering 'third_party/nlohmann' 2024-08-20T22:13:22.1253200Z Entering 'third_party/onnx' 2024-08-20T22:13:22.1322594Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:22.1372837Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:22.1427992Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:22.1481191Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:22.1530655Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:22.1580924Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:22.1632791Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:22.1684398Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:22.1733516Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:22.1783504Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:22.1831738Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:22.1884518Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:22.1937472Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:22.2013729Z Entering 'third_party/pocketfft' 2024-08-20T22:13:22.2066051Z Entering 'third_party/protobuf' 2024-08-20T22:13:22.2122801Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:22.2172945Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:22.2229074Z Entering 'third_party/psimd' 2024-08-20T22:13:22.2282291Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:22.2334539Z Entering 'third_party/pybind11' 2024-08-20T22:13:22.2387440Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:22.2439194Z Entering 'third_party/sleef' 2024-08-20T22:13:22.2491185Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:22.2542363Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:22.2592509Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:22.2641897Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:22.2692681Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:22.2740527Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:22.2818607Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:22.3180631Z Entering 'android/libs/fbjni' 2024-08-20T22:13:22.3232663Z Entering 'third_party/FP16' 2024-08-20T22:13:22.3285619Z Entering 'third_party/FXdiv' 2024-08-20T22:13:22.3338541Z Entering 'third_party/NNPACK' 2024-08-20T22:13:22.3391094Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:22.3445898Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:22.3515733Z Entering 'third_party/benchmark' 2024-08-20T22:13:22.3567957Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:22.3620993Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:22.3674256Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:22.3727286Z Entering 'third_party/cutlass' 2024-08-20T22:13:22.3787712Z Entering 'third_party/eigen' 2024-08-20T22:13:22.3842046Z Entering 'third_party/fbgemm' 2024-08-20T22:13:22.3894699Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:22.3947085Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:22.4003008Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:22.4060101Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:22.4114972Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:22.4168194Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:22.4224049Z Entering 'third_party/fmt' 2024-08-20T22:13:22.4276684Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:22.4328970Z Entering 'third_party/gloo' 2024-08-20T22:13:22.4381598Z Entering 'third_party/googletest' 2024-08-20T22:13:22.4434594Z Entering 'third_party/ideep' 2024-08-20T22:13:22.4485637Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:22.4546063Z Entering 'third_party/ittapi' 2024-08-20T22:13:22.4598855Z Entering 'third_party/kineto' 2024-08-20T22:13:22.4649952Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:22.4703468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:22.4755645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:22.4807094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:22.4858472Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:22.4911743Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:22.4966985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:22.5020834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:22.5071824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:22.5122888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:22.5176660Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:22.5228687Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:22.5286306Z Entering 'third_party/mimalloc' 2024-08-20T22:13:22.5340295Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:22.5392313Z Entering 'third_party/nlohmann' 2024-08-20T22:13:22.5445879Z Entering 'third_party/onnx' 2024-08-20T22:13:22.5513258Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:22.5564047Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:22.5623394Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:22.5677539Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:22.5726890Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:22.5776849Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:22.5830799Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:22.5881988Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:22.5931360Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:22.5981472Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:22.6029769Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:22.6082707Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:22.6135867Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:22.6214841Z Entering 'third_party/pocketfft' 2024-08-20T22:13:22.6268568Z Entering 'third_party/protobuf' 2024-08-20T22:13:22.6347338Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:22.6398129Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:22.6453001Z Entering 'third_party/psimd' 2024-08-20T22:13:22.6506979Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:22.6561285Z Entering 'third_party/pybind11' 2024-08-20T22:13:22.6614108Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:22.6666654Z Entering 'third_party/sleef' 2024-08-20T22:13:22.6719829Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:22.6772308Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:22.6824136Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:22.6873674Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:22.6923815Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:22.6972599Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:22.7048444Z ##[endgroup] 2024-08-20T22:13:22.7096744Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:22.7132339Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:22.7351101Z Prepare all required actions 2024-08-20T22:13:22.7351550Z Getting action download info 2024-08-20T22:13:22.9076250Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:22.9076677Z env: 2024-08-20T22:13:22.9076945Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.9077277Z ##[endgroup] 2024-08-20T22:13:22.9141680Z ##[group]Run set -euo pipefail 2024-08-20T22:13:22.9142122Z set -euo pipefail 2024-08-20T22:13:22.9142508Z function get_ec2_metadata() { 2024-08-20T22:13:22.9143032Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:22.9143905Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:22.9144739Z  category=$1 2024-08-20T22:13:22.9145271Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:22.9145900Z  runner_name_str=i-024c42853ba21bbc0 2024-08-20T22:13:22.9146385Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:22.9146872Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:22.9147427Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:22.9148102Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:22.9148695Z  else 2024-08-20T22:13:22.9149184Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:22.9149746Z  fi 2024-08-20T22:13:22.9150026Z } 2024-08-20T22:13:22.9150376Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:22.9150950Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:22.9151592Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:22.9152165Z echo "system info $(uname -a)" 2024-08-20T22:13:22.9164045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.9164539Z env: 2024-08-20T22:13:22.9164814Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.9165152Z ##[endgroup] 2024-08-20T22:13:22.9267294Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:22.9335845Z instance-id: i-024c42853ba21bbc0 2024-08-20T22:13:22.9404343Z instance-type: g5.4xlarge 2024-08-20T22:13:22.9418381Z system info Linux ip-10-0-56-249.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:22.9443444Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:22.9444343Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:22.9454360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.9455023Z env: 2024-08-20T22:13:22.9455292Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.9455630Z ##[endgroup] 2024-08-20T22:13:22.9534396Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:22.9534998Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:22.9535539Z  echo "Docker daemon is running..."; 2024-08-20T22:13:22.9535982Z else 2024-08-20T22:13:22.9536462Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:22.9537035Z fi 2024-08-20T22:13:22.9545443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.9545924Z env: 2024-08-20T22:13:22.9546194Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.9546527Z ##[endgroup] 2024-08-20T22:13:22.9634764Z Docker daemon is running... 2024-08-20T22:13:22.9688437Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:22.9688992Z with: 2024-08-20T22:13:22.9689272Z shell: bash 2024-08-20T22:13:22.9689558Z timeout_minutes: 5 2024-08-20T22:13:22.9689876Z max_attempts: 3 2024-08-20T22:13:22.9690184Z retry_wait_seconds: 30 2024-08-20T22:13:22.9693296Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:22.9696303Z polling_interval_seconds: 1 2024-08-20T22:13:22.9696675Z warning_on_retry: true 2024-08-20T22:13:22.9697017Z continue_on_error: false 2024-08-20T22:13:22.9697354Z env: 2024-08-20T22:13:22.9697620Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.9697967Z AWS_RETRY_MODE: standard 2024-08-20T22:13:22.9698305Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:22.9698643Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:22.9699014Z ##[endgroup] 2024-08-20T22:13:24.1246407Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:24.1247227Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:24.1248073Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:24.1248587Z 2024-08-20T22:13:24.1248704Z Login Succeeded 2024-08-20T22:13:25.0254473Z Command completed after 1 attempt(s). 2024-08-20T22:13:25.0319634Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:25.0320348Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:25.0320985Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:25.0330567Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.0331059Z env: 2024-08-20T22:13:25.0331328Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.0331668Z ##[endgroup] 2024-08-20T22:13:25.0427559Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:25.0428319Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:25.0428898Z # shellcheck disable=SC2046 2024-08-20T22:13:25.0429348Z docker stop $(docker ps -q) || true 2024-08-20T22:13:25.0429817Z # Prune all of the docker images 2024-08-20T22:13:25.0430263Z docker system prune -af 2024-08-20T22:13:25.0439365Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.0439858Z env: 2024-08-20T22:13:25.0440128Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.0440468Z ##[endgroup] 2024-08-20T22:13:25.0757155Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:25.0758380Z See 'docker stop --help'. 2024-08-20T22:13:25.0758619Z 2024-08-20T22:13:25.0758840Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:25.0759211Z 2024-08-20T22:13:25.0759357Z Stop one or more running containers 2024-08-20T22:13:25.1204035Z Total reclaimed space: 0B 2024-08-20T22:13:25.1248818Z ##[group]Run set +e 2024-08-20T22:13:25.1249166Z set +e 2024-08-20T22:13:25.1249461Z set -x 2024-08-20T22:13:25.1249749Z  2024-08-20T22:13:25.1250062Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:25.1250839Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:25.1251897Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:25.1252629Z # one is returned at random 2024-08-20T22:13:25.1253163Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:25.1253661Z  2024-08-20T22:13:25.1253983Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:25.1254594Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:25.1255537Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:25.1256083Z  2024-08-20T22:13:25.1256403Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:25.1256948Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:25.1257451Z  exit 1 2024-08-20T22:13:25.1257753Z  fi 2024-08-20T22:13:25.1258036Z fi 2024-08-20T22:13:25.1258312Z  2024-08-20T22:13:25.1258665Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:25.1259168Z  # Clean up any old records first 2024-08-20T22:13:25.1259666Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:25.1260113Z fi 2024-08-20T22:13:25.1260382Z  2024-08-20T22:13:25.1260811Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:25.1261348Z cat /etc/hosts 2024-08-20T22:13:25.1270473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.1270955Z env: 2024-08-20T22:13:25.1271237Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.1271579Z ##[endgroup] 2024-08-20T22:13:25.1300909Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:25.1307408Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:25.1307870Z ++ tail -n1 2024-08-20T22:13:25.1832748Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:25.1833248Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:25.1833685Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:25.1851868Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:25.1852626Z + sudo tee -a /etc/hosts 2024-08-20T22:13:25.4250810Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:25.4276058Z + cat /etc/hosts 2024-08-20T22:13:25.4286360Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:25.4298491Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:25.4299352Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:25.4456237Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:25.4456840Z with: 2024-08-20T22:13:25.4457950Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4459184Z docker-build-dir: .ci/docker 2024-08-20T22:13:25.4459562Z working-directory: . 2024-08-20T22:13:25.4460031Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:25.4460551Z force-push: false 2024-08-20T22:13:25.4460859Z env: 2024-08-20T22:13:25.4461135Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.4461470Z ##[endgroup] 2024-08-20T22:13:25.4480656Z ##[group]Run set -ex 2024-08-20T22:13:25.4481018Z set -ex 2024-08-20T22:13:25.4481311Z  2024-08-20T22:13:25.4482042Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:25.4482997Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:25.4483787Z # job could then download the pre-built image as usual 2024-08-20T22:13:25.4484523Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:25.4485189Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4485815Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4486369Z  2024-08-20T22:13:25.4486868Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:25.4487491Z  exit 0 2024-08-20T22:13:25.4487798Z else 2024-08-20T22:13:25.4488158Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4488612Z fi 2024-08-20T22:13:25.4488888Z  2024-08-20T22:13:25.4489352Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:25.4490185Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:25.4490951Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:25.4491644Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:25.4492359Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4493049Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4493594Z else 2024-08-20T22:13:25.4494026Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:25.4494682Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4495563Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4496300Z fi 2024-08-20T22:13:25.4508058Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.4508552Z env: 2024-08-20T22:13:25.4508820Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.4509160Z REPO_NAME: pytorch 2024-08-20T22:13:25.4510300Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4511519Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:25.4512008Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:25.4512514Z ##[endgroup] 2024-08-20T22:13:25.4543945Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:25.4544892Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:25.4545437Z + echo skip=false 2024-08-20T22:13:25.4547549Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:25.4551231Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4552845Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:25.4578850Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4579983Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4581882Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4617275Z ##[group]Run set +e 2024-08-20T22:13:25.4617809Z set +e 2024-08-20T22:13:25.4618190Z set -x 2024-08-20T22:13:25.4618628Z  2024-08-20T22:13:25.4619191Z login() { 2024-08-20T22:13:25.4620008Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:25.4620777Z } 2024-08-20T22:13:25.4621162Z  2024-08-20T22:13:25.4621588Z retry () { 2024-08-20T22:13:25.4622087Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:25.4622584Z } 2024-08-20T22:13:25.4622995Z  2024-08-20T22:13:25.4623418Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:25.4623879Z  2024-08-20T22:13:25.4624609Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:25.4625425Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:25.4626009Z  exit 0 2024-08-20T22:13:25.4626417Z fi 2024-08-20T22:13:25.4626798Z  2024-08-20T22:13:25.4627379Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:25.4628361Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:25.4629146Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:25.4629901Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:25.4630649Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:25.4631160Z else 2024-08-20T22:13:25.4631675Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:25.4632520Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:25.4633112Z fi 2024-08-20T22:13:25.4633496Z  2024-08-20T22:13:25.4633918Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:25.4634513Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4635080Z  2024-08-20T22:13:25.4635878Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:25.4636692Z  exit 0 2024-08-20T22:13:25.4637118Z fi 2024-08-20T22:13:25.4637538Z  2024-08-20T22:13:25.4638071Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:25.4639113Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:25.4640058Z  exit 1 2024-08-20T22:13:25.4640468Z fi 2024-08-20T22:13:25.4640883Z  2024-08-20T22:13:25.4641424Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:25.4642483Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:25.4643485Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:25.4644529Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:25.4645735Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:25.4646649Z fi 2024-08-20T22:13:25.4647041Z  2024-08-20T22:13:25.4647479Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:25.4657121Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.4657670Z env: 2024-08-20T22:13:25.4658099Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.4658574Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:25.4659123Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:25.4660512Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4661844Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:25.4662526Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:25.4663287Z ##[endgroup] 2024-08-20T22:13:25.4695096Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:25.4695852Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:25.4697926Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:25.4698970Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.0022891Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:26.0023972Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:26.0025210Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:26.0025825Z 2024-08-20T22:13:26.0025975Z Login Succeeded 2024-08-20T22:13:26.0049336Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.2112862Z { 2024-08-20T22:13:26.2113354Z "schemaVersion": 2, 2024-08-20T22:13:26.2114413Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:26.2115723Z "config": { 2024-08-20T22:13:26.2116634Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:26.2117537Z "size": 48711, 2024-08-20T22:13:26.2118942Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:26.2120076Z }, 2024-08-20T22:13:26.2120630Z "layers": [ 2024-08-20T22:13:26.2121059Z { 2024-08-20T22:13:26.2122156Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2123210Z "size": 28580681, 2024-08-20T22:13:26.2123856Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:26.2125042Z }, 2024-08-20T22:13:26.2125386Z { 2024-08-20T22:13:26.2126144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2127839Z "size": 7943451, 2024-08-20T22:13:26.2128577Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:26.2129382Z }, 2024-08-20T22:13:26.2129734Z { 2024-08-20T22:13:26.2130225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2131355Z "size": 55728572, 2024-08-20T22:13:26.2132005Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:26.2133387Z }, 2024-08-20T22:13:26.2133689Z { 2024-08-20T22:13:26.2134421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2135401Z "size": 186, 2024-08-20T22:13:26.2136082Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:26.2136979Z }, 2024-08-20T22:13:26.2137851Z { 2024-08-20T22:13:26.2138538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2139403Z "size": 6886, 2024-08-20T22:13:26.2141029Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:26.2141997Z }, 2024-08-20T22:13:26.2142752Z { 2024-08-20T22:13:26.2143468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2144293Z "size": 1329001756, 2024-08-20T22:13:26.2145124Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:26.2146017Z }, 2024-08-20T22:13:26.2146449Z { 2024-08-20T22:13:26.2146976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2147630Z "size": 62501, 2024-08-20T22:13:26.2148288Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:26.2148992Z }, 2024-08-20T22:13:26.2149387Z { 2024-08-20T22:13:26.2149886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2150518Z "size": 1684, 2024-08-20T22:13:26.2151303Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:26.2152199Z }, 2024-08-20T22:13:26.2152507Z { 2024-08-20T22:13:26.2153193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2153834Z "size": 1523, 2024-08-20T22:13:26.2154438Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:26.2155248Z }, 2024-08-20T22:13:26.2155580Z { 2024-08-20T22:13:26.2156118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2156788Z "size": 2528295403, 2024-08-20T22:13:26.2157448Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:26.2158180Z }, 2024-08-20T22:13:26.2158571Z { 2024-08-20T22:13:26.2159045Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2159705Z "size": 86016, 2024-08-20T22:13:26.2160465Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:26.2161189Z }, 2024-08-20T22:13:26.2161516Z { 2024-08-20T22:13:26.2162095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2162754Z "size": 1822, 2024-08-20T22:13:26.2163353Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:26.2164102Z }, 2024-08-20T22:13:26.2164464Z { 2024-08-20T22:13:26.2165034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2165631Z "size": 246761563, 2024-08-20T22:13:26.2166315Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:26.2167159Z }, 2024-08-20T22:13:26.2167494Z { 2024-08-20T22:13:26.2168004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2168748Z "size": 544, 2024-08-20T22:13:26.2169376Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:26.2170039Z }, 2024-08-20T22:13:26.2170442Z { 2024-08-20T22:13:26.2170973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2171610Z "size": 1258, 2024-08-20T22:13:26.2172639Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:26.2173370Z }, 2024-08-20T22:13:26.2173711Z { 2024-08-20T22:13:26.2174360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2174982Z "size": 484, 2024-08-20T22:13:26.2175601Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:26.2176367Z }, 2024-08-20T22:13:26.2176665Z { 2024-08-20T22:13:26.2177181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2177889Z "size": 91711766, 2024-08-20T22:13:26.2178599Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:26.2179282Z }, 2024-08-20T22:13:26.2179710Z { 2024-08-20T22:13:26.2180251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2181007Z "size": 3390, 2024-08-20T22:13:26.2182889Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:26.2183602Z }, 2024-08-20T22:13:26.2183952Z { 2024-08-20T22:13:26.2184616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2185211Z "size": 1909, 2024-08-20T22:13:26.2185845Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:26.2186592Z }, 2024-08-20T22:13:26.2186888Z { 2024-08-20T22:13:26.2187430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2188116Z "size": 701, 2024-08-20T22:13:26.2188774Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:26.2189528Z }, 2024-08-20T22:13:26.2189912Z { 2024-08-20T22:13:26.2190575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2191271Z "size": 2864254879, 2024-08-20T22:13:26.2191903Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:26.2192627Z }, 2024-08-20T22:13:26.2193032Z { 2024-08-20T22:13:26.2193538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2194173Z "size": 380, 2024-08-20T22:13:26.2194840Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:26.2195550Z }, 2024-08-20T22:13:26.2195923Z { 2024-08-20T22:13:26.2196491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2197157Z "size": 12900, 2024-08-20T22:13:26.2197795Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:26.2198518Z }, 2024-08-20T22:13:26.2198931Z { 2024-08-20T22:13:26.2199414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2200126Z "size": 803, 2024-08-20T22:13:26.2200757Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:26.2201446Z }, 2024-08-20T22:13:26.2201814Z { 2024-08-20T22:13:26.2202399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2203061Z "size": 106, 2024-08-20T22:13:26.2203694Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:26.2204379Z }, 2024-08-20T22:13:26.2204734Z { 2024-08-20T22:13:26.2205305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2205905Z "size": 503, 2024-08-20T22:13:26.2206548Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:26.2207291Z }, 2024-08-20T22:13:26.2207624Z { 2024-08-20T22:13:26.2208124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2208819Z "size": 121477267, 2024-08-20T22:13:26.2209550Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:26.2210295Z }, 2024-08-20T22:13:26.2210595Z { 2024-08-20T22:13:26.2211131Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2211835Z "size": 109, 2024-08-20T22:13:26.2212429Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:26.2213147Z }, 2024-08-20T22:13:26.2213533Z { 2024-08-20T22:13:26.2214048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2214665Z "size": 490, 2024-08-20T22:13:26.2215333Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:26.2216127Z }, 2024-08-20T22:13:26.2216462Z { 2024-08-20T22:13:26.2216989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2217644Z "size": 297, 2024-08-20T22:13:26.2218264Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:26.2218968Z }, 2024-08-20T22:13:26.2219425Z { 2024-08-20T22:13:26.2219935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2220640Z "size": 103, 2024-08-20T22:13:26.2221225Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:26.2221923Z }, 2024-08-20T22:13:26.2222397Z { 2024-08-20T22:13:26.2222905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2223501Z "size": 1473, 2024-08-20T22:13:26.2224189Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:26.2224992Z }, 2024-08-20T22:13:26.2225291Z { 2024-08-20T22:13:26.2225875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2226507Z "size": 594729516, 2024-08-20T22:13:26.2227158Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:26.2227979Z }, 2024-08-20T22:13:26.2228318Z { 2024-08-20T22:13:26.2228860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2229647Z "size": 164, 2024-08-20T22:13:26.2230232Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:26.2230950Z }, 2024-08-20T22:13:26.2231325Z { 2024-08-20T22:13:26.2231851Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2232441Z "size": 564, 2024-08-20T22:13:26.2233101Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:26.2233820Z }, 2024-08-20T22:13:26.2234188Z { 2024-08-20T22:13:26.2234776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2235411Z "size": 35874203, 2024-08-20T22:13:26.2236190Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:26.2236898Z }, 2024-08-20T22:13:26.2237241Z { 2024-08-20T22:13:26.2237822Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2238528Z "size": 104, 2024-08-20T22:13:26.2239104Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:26.2239846Z }, 2024-08-20T22:13:26.2240208Z { 2024-08-20T22:13:26.2240685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2241392Z "size": 425, 2024-08-20T22:13:26.2242037Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:26.2242822Z }, 2024-08-20T22:13:26.2243188Z { 2024-08-20T22:13:26.2243705Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2244352Z "size": 20262183, 2024-08-20T22:13:26.2245062Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:26.2245729Z }, 2024-08-20T22:13:26.2246069Z { 2024-08-20T22:13:26.2246662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2247313Z "size": 438, 2024-08-20T22:13:26.2247919Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:26.2248675Z }, 2024-08-20T22:13:26.2249063Z { 2024-08-20T22:13:26.2249635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2250332Z "size": 701, 2024-08-20T22:13:26.2250964Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:26.2251696Z }, 2024-08-20T22:13:26.2252052Z { 2024-08-20T22:13:26.2252573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2253232Z "size": 141, 2024-08-20T22:13:26.2253928Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:26.2254620Z }, 2024-08-20T22:13:26.2254965Z { 2024-08-20T22:13:26.2255526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2256257Z "size": 135, 2024-08-20T22:13:26.2256931Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:26.2257708Z }, 2024-08-20T22:13:26.2258160Z + exit 0 2024-08-20T22:13:26.2258575Z { 2024-08-20T22:13:26.2259174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2259786Z "size": 4838068375, 2024-08-20T22:13:26.2260483Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:26.2261193Z }, 2024-08-20T22:13:26.2261527Z { 2024-08-20T22:13:26.2262079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2262842Z "size": 194, 2024-08-20T22:13:26.2263511Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:26.2264337Z }, 2024-08-20T22:13:26.2264863Z { 2024-08-20T22:13:26.2265382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2266099Z "size": 564, 2024-08-20T22:13:26.2266699Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:26.2267423Z }, 2024-08-20T22:13:26.2267812Z { 2024-08-20T22:13:26.2268333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2268947Z "size": 43162921, 2024-08-20T22:13:26.2269624Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:26.2270390Z }, 2024-08-20T22:13:26.2270744Z { 2024-08-20T22:13:26.2271265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2271898Z "size": 106, 2024-08-20T22:13:26.2272785Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:26.2273548Z }, 2024-08-20T22:13:26.2273866Z { 2024-08-20T22:13:26.2274373Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2275068Z "size": 1380, 2024-08-20T22:13:26.2275670Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:26.2276349Z }, 2024-08-20T22:13:26.2276822Z { 2024-08-20T22:13:26.2277370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2277970Z "size": 701, 2024-08-20T22:13:26.2278699Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:26.2279424Z }, 2024-08-20T22:13:26.2279817Z { 2024-08-20T22:13:26.2280319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2280958Z "size": 139, 2024-08-20T22:13:26.2281638Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:26.2282311Z }, 2024-08-20T22:13:26.2282650Z { 2024-08-20T22:13:26.2283219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2283961Z "size": 118, 2024-08-20T22:13:26.2284549Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:26.2285285Z }, 2024-08-20T22:13:26.2285641Z { 2024-08-20T22:13:26.2286167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2286840Z "size": 4324672728, 2024-08-20T22:13:26.2287506Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:26.2288219Z }, 2024-08-20T22:13:26.2288623Z { 2024-08-20T22:13:26.2289106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2289747Z "size": 174, 2024-08-20T22:13:26.2290506Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:26.2291159Z }, 2024-08-20T22:13:26.2291499Z { 2024-08-20T22:13:26.2292090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2292738Z "size": 907, 2024-08-20T22:13:26.2293358Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:26.2294289Z }, 2024-08-20T22:13:26.2294633Z { 2024-08-20T22:13:26.2295177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2295835Z "size": 701, 2024-08-20T22:13:26.2296482Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:26.2297261Z }, 2024-08-20T22:13:26.2297614Z { 2024-08-20T22:13:26.2298178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2298838Z "size": 134, 2024-08-20T22:13:26.2299520Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:26.2300204Z }, 2024-08-20T22:13:26.2300542Z { 2024-08-20T22:13:26.2301116Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2303314Z "size": 32, 2024-08-20T22:13:26.2304042Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:26.2304962Z }, 2024-08-20T22:13:26.2305384Z { 2024-08-20T22:13:26.2305868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2306602Z "size": 158, 2024-08-20T22:13:26.2307238Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:26.2308028Z }, 2024-08-20T22:13:26.2308328Z { 2024-08-20T22:13:26.2308857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2309571Z "size": 1841, 2024-08-20T22:13:26.2310212Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:26.2310879Z }, 2024-08-20T22:13:26.2311291Z { 2024-08-20T22:13:26.2311819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2312537Z "size": 7529776, 2024-08-20T22:13:26.2313221Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:26.2313943Z }, 2024-08-20T22:13:26.2314307Z { 2024-08-20T22:13:26.2314854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2315499Z "size": 163, 2024-08-20T22:13:26.2316149Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:26.2316901Z }, 2024-08-20T22:13:26.2317205Z { 2024-08-20T22:13:26.2317754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2318463Z "size": 7943, 2024-08-20T22:13:26.2319171Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:26.2319859Z }, 2024-08-20T22:13:26.2320266Z { 2024-08-20T22:13:26.2320808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2321452Z "size": 8068, 2024-08-20T22:13:26.2322100Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:26.2322830Z }, 2024-08-20T22:13:26.2323160Z { 2024-08-20T22:13:26.2323696Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2324352Z "size": 302, 2024-08-20T22:13:26.2324965Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:26.2325703Z }, 2024-08-20T22:13:26.2326099Z { 2024-08-20T22:13:26.2326624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2327335Z "size": 7629638, 2024-08-20T22:13:26.2327967Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:26.2328618Z }, 2024-08-20T22:13:26.2329022Z { 2024-08-20T22:13:26.2329539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2330138Z "size": 108, 2024-08-20T22:13:26.2330844Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:26.2331563Z }, 2024-08-20T22:13:26.2331970Z { 2024-08-20T22:13:26.2332478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2333294Z "size": 54145661, 2024-08-20T22:13:26.2334029Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:26.2334736Z }, 2024-08-20T22:13:26.2335029Z { 2024-08-20T22:13:26.2335609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2336241Z "size": 473, 2024-08-20T22:13:26.2336886Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:26.2337606Z }, 2024-08-20T22:13:26.2337940Z { 2024-08-20T22:13:26.2338482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2339127Z "size": 1374858766, 2024-08-20T22:13:26.2339838Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:26.2340544Z }, 2024-08-20T22:13:26.2340938Z { 2024-08-20T22:13:26.2341496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2342147Z "size": 106, 2024-08-20T22:13:26.2342835Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:26.2343564Z }, 2024-08-20T22:13:26.2343855Z { 2024-08-20T22:13:26.2344416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2345154Z "size": 559, 2024-08-20T22:13:26.2345722Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:26.2346530Z }, 2024-08-20T22:13:26.2346881Z { 2024-08-20T22:13:26.2347384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2348024Z "size": 46248488, 2024-08-20T22:13:26.2348677Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:26.2349363Z }, 2024-08-20T22:13:26.2349759Z { 2024-08-20T22:13:26.2350234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2350877Z "size": 111, 2024-08-20T22:13:26.2351567Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:26.2352252Z }, 2024-08-20T22:13:26.2352746Z { 2024-08-20T22:13:26.2353251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2353877Z "size": 32, 2024-08-20T22:13:26.2354517Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:26.2355218Z }, 2024-08-20T22:13:26.2355543Z { 2024-08-20T22:13:26.2356125Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2356716Z "size": 32, 2024-08-20T22:13:26.2357342Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:26.2358127Z }, 2024-08-20T22:13:26.2358463Z { 2024-08-20T22:13:26.2358966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2359735Z "size": 32, 2024-08-20T22:13:26.2360359Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:26.2361079Z }, 2024-08-20T22:13:26.2361444Z { 2024-08-20T22:13:26.2361965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:26.2362635Z "size": 32, 2024-08-20T22:13:26.2363286Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:26.2363996Z } 2024-08-20T22:13:26.2364361Z ] 2024-08-20T22:13:26.2364755Z } 2024-08-20T22:13:26.2658971Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:26.2659474Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:26.2660014Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:26.2669254Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.2669748Z env: 2024-08-20T22:13:26.2670018Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.2671190Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.2672619Z ##[endgroup] 2024-08-20T22:13:26.2705458Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.2757508Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:26.2758107Z with: 2024-08-20T22:13:26.2759234Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.2760560Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.2761080Z env: 2024-08-20T22:13:26.2761357Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.2761705Z ##[endgroup] 2024-08-20T22:13:26.2779052Z ##[group]Run set -x 2024-08-20T22:13:26.2779546Z set -x 2024-08-20T22:13:26.2779848Z set +e 2024-08-20T22:13:26.2780141Z  2024-08-20T22:13:26.2780413Z login() { 2024-08-20T22:13:26.2781071Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:26.2781783Z } 2024-08-20T22:13:26.2782063Z  2024-08-20T22:13:26.2782367Z retry () { 2024-08-20T22:13:26.2782759Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:26.2783202Z } 2024-08-20T22:13:26.2783481Z  2024-08-20T22:13:26.2783798Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:26.2784211Z  2024-08-20T22:13:26.2784566Z set -e 2024-08-20T22:13:26.2785041Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:26.2785740Z # only pull docker image if it's not available locally 2024-08-20T22:13:26.2786500Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:26.2787201Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:26.2787643Z fi 2024-08-20T22:13:26.2795630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:26.2796122Z env: 2024-08-20T22:13:26.2796400Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:26.2797561Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.2798884Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.2799403Z ##[endgroup] 2024-08-20T22:13:26.2826192Z + set +e 2024-08-20T22:13:26.2826714Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.2827346Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.2830228Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:26.2831398Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:26.8121605Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:26.8122686Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:26.8123669Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:26.8124218Z 2024-08-20T22:13:26.8124341Z Login Succeeded 2024-08-20T22:13:26.8143184Z + set -e 2024-08-20T22:13:26.8145042Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8305893Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:26.8308095Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:27.0575783Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:27.0578232Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:27.0578791Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:27.0580912Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:27.0581438Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:27.0581955Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:27.0583490Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:27.0584241Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:27.0584911Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:27.0585417Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:27.0585909Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:27.0586410Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:27.0586906Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:27.0587412Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:27.0588141Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:27.0588636Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:27.0589134Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:27.0589628Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:27.0590109Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:27.0590600Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:27.0591098Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:27.0591578Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:27.0592074Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:27.0592564Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:27.0593048Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:27.0593528Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:27.0594036Z b57676e46aee: Waiting 2024-08-20T22:13:27.0594479Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:27.0594859Z a8c1e85b5e14: Waiting 2024-08-20T22:13:27.0595189Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:27.0595537Z 0bf61114d37b: Waiting 2024-08-20T22:13:27.0595877Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:27.0596235Z 2df849f72ba4: Waiting 2024-08-20T22:13:27.0596559Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:27.0596929Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:27.0597283Z a41a8d1c11c8: Waiting 2024-08-20T22:13:27.0597601Z d527cbbb87e3: Waiting 2024-08-20T22:13:27.0597911Z ca426ffc9d2b: Waiting 2024-08-20T22:13:27.0598231Z ec1e7978c1fe: Waiting 2024-08-20T22:13:27.0598549Z 75b2abb16c68: Waiting 2024-08-20T22:13:27.0598862Z 4373aa743caf: Waiting 2024-08-20T22:13:27.0599165Z 0c1227890755: Waiting 2024-08-20T22:13:27.0599477Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:27.0599791Z d8d1234baab3: Waiting 2024-08-20T22:13:27.0600098Z 7ed32bc8e469: Waiting 2024-08-20T22:13:27.0600412Z b119c4fc6c58: Waiting 2024-08-20T22:13:27.0600724Z d3547bd43146: Waiting 2024-08-20T22:13:27.0601032Z 5645520256fa: Waiting 2024-08-20T22:13:27.0601347Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:27.0601719Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:27.0602094Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:27.0602456Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:27.0602827Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:27.0603205Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:27.0603577Z 019620471c25: Pulling fs layer 2024-08-20T22:13:27.0603938Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:27.0604298Z 1fef4bd19792: Waiting 2024-08-20T22:13:27.0604636Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:27.0604989Z 33444a5a7e01: Waiting 2024-08-20T22:13:27.0605296Z 7cbfce999d76: Waiting 2024-08-20T22:13:27.0605621Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:27.0605980Z 6efb96fc87ea: Waiting 2024-08-20T22:13:27.0606304Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:27.0606659Z 0479fbac3361: Waiting 2024-08-20T22:13:27.0606988Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:27.0607347Z f1d6874dee9c: Waiting 2024-08-20T22:13:27.0607670Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:27.0608045Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:27.0608423Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:27.0608780Z afc429e87b59: Waiting 2024-08-20T22:13:27.0609102Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:27.0609466Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:27.0609815Z 019620471c25: Waiting 2024-08-20T22:13:27.0610119Z 829c0e19f4b7: Waiting 2024-08-20T22:13:27.0610448Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:27.0610934Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:27.0611260Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:27.0611616Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:27.0611969Z 560cd6122a38: Waiting 2024-08-20T22:13:27.0612293Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:27.0612655Z f957c1624481: Pulling fs layer 2024-08-20T22:13:27.0613022Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:27.0613369Z 4a0412373305: Waiting 2024-08-20T22:13:27.0613673Z 302c8c177944: Waiting 2024-08-20T22:13:27.0613973Z 02970347dff6: Waiting 2024-08-20T22:13:27.0614299Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:27.0614659Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:27.0615070Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:27.0615431Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:27.0615785Z 385d0421d67c: Waiting 2024-08-20T22:13:27.0616105Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:27.0616451Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:27.0616776Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:27.0617146Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:27.0617513Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:27.0617873Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:27.0618244Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:27.0618610Z ee1273153675: Pulling fs layer 2024-08-20T22:13:27.0618973Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:27.0619333Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:27.0619691Z 7754ee068b33: Waiting 2024-08-20T22:13:27.0620000Z 5a4eb19d7219: Waiting 2024-08-20T22:13:27.0620319Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:27.0620686Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:27.0621036Z 1811ff4d3556: Waiting 2024-08-20T22:13:27.0621352Z c42b3602a1fb: Waiting 2024-08-20T22:13:27.0621672Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:27.0622026Z b10de02255c0: Waiting 2024-08-20T22:13:27.0622334Z f957c1624481: Waiting 2024-08-20T22:13:27.0622656Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:27.0623017Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:27.0623376Z cbb79a705353: Waiting 2024-08-20T22:13:27.0623700Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:27.0624045Z e56eeff2db93: Waiting 2024-08-20T22:13:27.0624361Z 11fad972ef6c: Waiting 2024-08-20T22:13:27.0624783Z ee1273153675: Waiting 2024-08-20T22:13:27.0625089Z d79fd7d52698: Waiting 2024-08-20T22:13:27.0625390Z 7c47122f4e09: Waiting 2024-08-20T22:13:27.0625698Z 5d115b29b5d5: Waiting 2024-08-20T22:13:27.0626007Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:27.0626313Z eb09cbf3e416: Waiting 2024-08-20T22:13:27.0626626Z da6a838a66c7: Waiting 2024-08-20T22:13:27.0626936Z 37513e9b43d7: Waiting 2024-08-20T22:13:27.0627245Z a7a5392e2bf6: Waiting 2024-08-20T22:13:27.0627555Z 78ee77eddef2: Waiting 2024-08-20T22:13:27.0627863Z 768c31dca016: Waiting 2024-08-20T22:13:27.0628179Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:27.0628485Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:27.0628797Z 83808b2eff4d: Waiting 2024-08-20T22:13:27.0629107Z 4f4fb700ef54: Waiting 2024-08-20T22:13:27.0629415Z 916c5e858016: Waiting 2024-08-20T22:13:27.0629725Z 9752d6fbbcab: Waiting 2024-08-20T22:13:27.0630032Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:27.0630342Z e22979f9e0ee: Waiting 2024-08-20T22:13:27.1951714Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:27.1952318Z 224fe954d725: Download complete 2024-08-20T22:13:27.2891147Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:27.2891685Z d527cbbb87e3: Download complete 2024-08-20T22:13:27.3601975Z b57676e46aee: Verifying Checksum 2024-08-20T22:13:27.3602524Z b57676e46aee: Download complete 2024-08-20T22:13:27.4056185Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:27.4056718Z 7a2c55901189: Download complete 2024-08-20T22:13:27.4737259Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:27.4737883Z a41a8d1c11c8: Download complete 2024-08-20T22:13:27.5379654Z 0c1227890755: Download complete 2024-08-20T22:13:27.6175922Z d8d1234baab3: Download complete 2024-08-20T22:13:27.6738729Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:27.6739236Z 75722010b82e: Download complete 2024-08-20T22:13:27.7536176Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:27.7536737Z ec1e7978c1fe: Download complete 2024-08-20T22:13:27.8315324Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:27.8316042Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:28.5708481Z 7a2c55901189: Pull complete 2024-08-20T22:13:28.8981317Z 224fe954d725: Pull complete 2024-08-20T22:13:29.8201536Z 75722010b82e: Pull complete 2024-08-20T22:13:29.8414542Z d527cbbb87e3: Pull complete 2024-08-20T22:13:29.8647766Z b57676e46aee: Pull complete 2024-08-20T22:13:30.3482319Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:30.3482894Z ca426ffc9d2b: Download complete 2024-08-20T22:13:30.4193201Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:30.4193631Z 4373aa743caf: Download complete 2024-08-20T22:13:30.4893470Z 75b2abb16c68: Download complete 2024-08-20T22:13:30.6058001Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:31.5800696Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:31.5801266Z 0bf61114d37b: Download complete 2024-08-20T22:13:31.6501252Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:31.6501707Z 2df849f72ba4: Download complete 2024-08-20T22:13:31.7299723Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:31.8111049Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:13:31.8111538Z 1fef4bd19792: Download complete 2024-08-20T22:13:40.7324341Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:40.7324992Z a8c1e85b5e14: Download complete 2024-08-20T22:13:40.7905453Z b119c4fc6c58: Download complete 2024-08-20T22:13:40.8573854Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:40.8574467Z 33b639226eee: Download complete 2024-08-20T22:13:40.9205354Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:40.9205951Z d3547bd43146: Download complete 2024-08-20T22:13:40.9902154Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:40.9902726Z 5645520256fa: Download complete 2024-08-20T22:13:41.0731434Z 829ed7d91e38: Download complete 2024-08-20T22:13:42.3350510Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:42.3351183Z 385d0421d67c: Download complete 2024-08-20T22:13:42.4027840Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:42.4028407Z 7cbfce999d76: Download complete 2024-08-20T22:13:42.4805746Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:42.4806322Z 6658e558ebe4: Download complete 2024-08-20T22:13:42.5595092Z 33444a5a7e01: Download complete 2024-08-20T22:13:42.6336141Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:42.6336584Z 7c47122f4e09: Download complete 2024-08-20T22:13:42.7189438Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:42.7190033Z 1811ff4d3556: Download complete 2024-08-20T22:13:48.7169090Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:48.7169689Z 7754ee068b33: Download complete 2024-08-20T22:13:48.8012595Z 0479fbac3361: Download complete 2024-08-20T22:13:48.8617175Z f1d6874dee9c: Download complete 2024-08-20T22:13:49.2978153Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:49.2978652Z afc429e87b59: Download complete 2024-08-20T22:13:49.3678155Z 019620471c25: Verifying Checksum 2024-08-20T22:13:49.3678687Z 019620471c25: Download complete 2024-08-20T22:13:49.4513533Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:49.4514175Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:49.6983282Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:49.6983761Z 829c0e19f4b7: Download complete 2024-08-20T22:13:49.7706343Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:49.7708156Z 560cd6122a38: Download complete 2024-08-20T22:13:49.8409295Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:49.8409883Z 5a4eb19d7219: Download complete 2024-08-20T22:13:49.9225607Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:49.9226286Z da6a838a66c7: Download complete 2024-08-20T22:13:52.9517919Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:52.9518434Z 7ed32bc8e469: Download complete 2024-08-20T22:13:53.1713928Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:53.1714460Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:53.2393029Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:53.2393614Z 0f085d4fa279: Download complete 2024-08-20T22:13:53.7169616Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:53.7170135Z 4a0412373305: Download complete 2024-08-20T22:13:53.7813583Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:53.7814023Z 302c8c177944: Download complete 2024-08-20T22:13:53.8498449Z 02970347dff6: Verifying Checksum 2024-08-20T22:13:53.8498921Z 02970347dff6: Download complete 2024-08-20T22:13:53.9159309Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:53.9840907Z 916c5e858016: Download complete 2024-08-20T22:13:53.9841363Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:53.9841758Z c42b3602a1fb: Download complete 2024-08-20T22:13:54.1381010Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:54.3341114Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:54.4642268Z 0c1227890755: Pull complete 2024-08-20T22:13:54.6345397Z d8d1234baab3: Pull complete 2024-08-20T22:14:00.5010986Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:14:00.5011622Z 6efb96fc87ea: Download complete 2024-08-20T22:14:00.5824567Z f957c1624481: Download complete 2024-08-20T22:14:00.6608671Z 768c31dca016: Verifying Checksum 2024-08-20T22:14:00.6609288Z 768c31dca016: Download complete 2024-08-20T22:14:00.7554151Z 8c32d0ecebbe: Download complete 2024-08-20T22:14:00.7649262Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:14:00.7649832Z 4f4fb700ef54: Download complete 2024-08-20T22:14:00.8341798Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:14:00.8342421Z 78ee77eddef2: Download complete 2024-08-20T22:14:00.8991162Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:14:00.8991752Z 37513e9b43d7: Download complete 2024-08-20T22:14:01.0328229Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:14:01.0328752Z a7a5392e2bf6: Download complete 2024-08-20T22:14:01.1145917Z b10de02255c0: Verifying Checksum 2024-08-20T22:14:01.1146388Z b10de02255c0: Download complete 2024-08-20T22:14:01.1924942Z e56eeff2db93: Download complete 2024-08-20T22:14:01.2616921Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:14:01.2617402Z 70ac3e38d0fd: Download complete 2024-08-20T22:14:01.3411332Z eb09cbf3e416: Download complete 2024-08-20T22:14:01.4874263Z ee1273153675: Verifying Checksum 2024-08-20T22:14:01.4874748Z ee1273153675: Download complete 2024-08-20T22:14:01.5636197Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:14:01.5636687Z e22979f9e0ee: Download complete 2024-08-20T22:14:02.1604012Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:14:02.1604636Z 6c1f0aabce7c: Download complete 2024-08-20T22:14:02.2319294Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:27.4687984Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:27.4688500Z 5d115b29b5d5: Download complete 2024-08-20T22:14:27.6102416Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:27.6102937Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:27.6836428Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:27.6836855Z d79fd7d52698: Download complete 2024-08-20T22:14:29.1635078Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:29.1635556Z 9752d6fbbcab: Download complete 2024-08-20T22:14:29.2294624Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:29.2295060Z 83808b2eff4d: Download complete 2024-08-20T22:15:08.8188649Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:15:08.8189299Z 11fad972ef6c: Download complete 2024-08-20T22:15:09.2683669Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:09.2684142Z cbb79a705353: Download complete 2024-08-20T22:15:24.0364405Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:24.2692464Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:24.4964962Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:33.1637462Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:33.3799628Z 4373aa743caf: Pull complete 2024-08-20T22:15:33.6004899Z 75b2abb16c68: Pull complete 2024-08-20T22:15:33.8236919Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:36.2659871Z 0bf61114d37b: Pull complete 2024-08-20T22:15:36.3932143Z 2df849f72ba4: Pull complete 2024-08-20T22:15:36.5501020Z 34c47d63c5de: Pull complete 2024-08-20T22:15:36.6862899Z 1fef4bd19792: Pull complete 2024-08-20T22:16:34.9894633Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:35.1973966Z b119c4fc6c58: Pull complete 2024-08-20T22:16:35.4449199Z 33b639226eee: Pull complete 2024-08-20T22:16:35.6696925Z d3547bd43146: Pull complete 2024-08-20T22:16:35.8971287Z 5645520256fa: Pull complete 2024-08-20T22:16:36.1224612Z 829ed7d91e38: Pull complete 2024-08-20T22:16:39.6123020Z 385d0421d67c: Pull complete 2024-08-20T22:16:39.7845948Z 7cbfce999d76: Pull complete 2024-08-20T22:16:39.9224937Z 6658e558ebe4: Pull complete 2024-08-20T22:16:40.1332567Z 33444a5a7e01: Pull complete 2024-08-20T22:16:40.3180283Z 7c47122f4e09: Pull complete 2024-08-20T22:16:40.5291853Z 1811ff4d3556: Pull complete 2024-08-20T22:16:50.7165723Z 7754ee068b33: Pull complete 2024-08-20T22:16:50.9044045Z 0479fbac3361: Pull complete 2024-08-20T22:16:51.1113097Z f1d6874dee9c: Pull complete 2024-08-20T22:16:52.0619116Z afc429e87b59: Pull complete 2024-08-20T22:16:52.2267313Z 019620471c25: Pull complete 2024-08-20T22:16:52.4289180Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:52.9285389Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:53.1373244Z 560cd6122a38: Pull complete 2024-08-20T22:16:53.5586856Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:53.7807468Z da6a838a66c7: Pull complete 2024-08-20T22:17:32.4333690Z cbb79a705353: Pull complete 2024-08-20T22:17:32.6484772Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:32.7909699Z 0f085d4fa279: Pull complete 2024-08-20T22:17:35.0868848Z 4a0412373305: Pull complete 2024-08-20T22:17:35.1108658Z 302c8c177944: Pull complete 2024-08-20T22:17:35.1308230Z 02970347dff6: Pull complete 2024-08-20T22:17:35.1783593Z 916c5e858016: Pull complete 2024-08-20T22:17:35.2004374Z c42b3602a1fb: Pull complete 2024-08-20T22:18:57.1314081Z 11fad972ef6c: Pull complete 2024-08-20T22:18:57.3561089Z f957c1624481: Pull complete 2024-08-20T22:18:57.5718747Z 768c31dca016: Pull complete 2024-08-20T22:18:57.9627247Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:58.1697416Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:58.3536777Z 78ee77eddef2: Pull complete 2024-08-20T22:18:58.5693468Z 37513e9b43d7: Pull complete 2024-08-20T22:18:58.9372288Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:59.0845263Z b10de02255c0: Pull complete 2024-08-20T22:18:59.3031501Z e56eeff2db93: Pull complete 2024-08-20T22:18:59.5193040Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:59.7393667Z eb09cbf3e416: Pull complete 2024-08-20T22:19:01.1323072Z ee1273153675: Pull complete 2024-08-20T22:19:01.3558725Z e22979f9e0ee: Pull complete 2024-08-20T22:19:03.5643498Z 6c1f0aabce7c: Pull complete 2024-08-20T22:19:03.7779050Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:16.9415122Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:17.0996586Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:17.2770450Z d79fd7d52698: Pull complete 2024-08-20T22:19:18.1310294Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:18.3283650Z 83808b2eff4d: Pull complete 2024-08-20T22:19:19.1964667Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:19.2416884Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:19.2642244Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:19.2739964Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:19.2740899Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:19.2751825Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:19.2752313Z env: 2024-08-20T22:19:19.2752586Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:19.2752940Z ##[endgroup] 2024-08-20T22:19:19.2966196Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:19.2966744Z with: 2024-08-20T22:19:19.2967021Z driver-version: 550.54.15 2024-08-20T22:19:19.2967350Z env: 2024-08-20T22:19:19.2967619Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:19.2967957Z ##[endgroup] 2024-08-20T22:19:19.3052180Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:19.3052727Z with: 2024-08-20T22:19:19.3053091Z timeout_minutes: 10 2024-08-20T22:19:19.3053423Z max_attempts: 3 2024-08-20T22:19:19.3086156Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:19.3119044Z retry_wait_seconds: 10 2024-08-20T22:19:19.3119410Z polling_interval_seconds: 1 2024-08-20T22:19:19.3119783Z warning_on_retry: true 2024-08-20T22:19:19.3120174Z continue_on_error: false 2024-08-20T22:19:19.3120508Z env: 2024-08-20T22:19:19.3120772Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:19.3121114Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:19.3121463Z ##[endgroup] 2024-08-20T22:19:19.3928074Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:19.3928996Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:19.3929600Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:19.8650764Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:19.8651308Z No packages marked for removal. 2024-08-20T22:19:19.8714741Z Dependencies resolved. 2024-08-20T22:19:19.8724890Z Nothing to do. 2024-08-20T22:19:19.8725362Z Complete! 2024-08-20T22:19:19.9543525Z + install_nvidia_driver_common 2024-08-20T22:19:19.9548814Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:19.9549466Z + lspci 2024-08-20T22:19:19.9549768Z Before installing NVIDIA driver 2024-08-20T22:19:20.0341224Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:20.0342040Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:20.0343299Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:20.0344495Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:20.0345608Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:20.0346591Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:20.0347322Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:20.0348135Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:20.0348729Z + lsmod 2024-08-20T22:19:20.0386853Z Module Size Used by 2024-08-20T22:19:20.0387289Z xt_conntrack 16384 1 2024-08-20T22:19:20.0387674Z nft_chain_nat 16384 3 2024-08-20T22:19:20.0388049Z xt_MASQUERADE 20480 1 2024-08-20T22:19:20.0388490Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:20.0388971Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:20.0389540Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:20.0390235Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:20.0390687Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:20.0391107Z xfrm_user 57344 1 2024-08-20T22:19:20.0391604Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:20.0392032Z xt_addrtype 16384 2 2024-08-20T22:19:20.0392398Z nft_compat 20480 4 2024-08-20T22:19:20.0392853Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:20.0394880Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:20.0395423Z br_netfilter 36864 0 2024-08-20T22:19:20.0395822Z bridge 307200 1 br_netfilter 2024-08-20T22:19:20.0396262Z stp 16384 1 bridge 2024-08-20T22:19:20.0396675Z llc 16384 2 bridge,stp 2024-08-20T22:19:20.0397085Z overlay 167936 0 2024-08-20T22:19:20.0397438Z tls 114688 0 2024-08-20T22:19:20.0397803Z nls_ascii 16384 1 2024-08-20T22:19:20.0398165Z nls_cp437 20480 1 2024-08-20T22:19:20.0398522Z vfat 24576 1 2024-08-20T22:19:20.0398874Z fat 86016 1 vfat 2024-08-20T22:19:20.0399257Z ena 167936 0 2024-08-20T22:19:20.0399623Z ptp 36864 1 ena 2024-08-20T22:19:20.0400007Z pps_core 24576 1 ptp 2024-08-20T22:19:20.0400400Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:20.0400774Z aesni_intel 393216 0 2024-08-20T22:19:20.0401144Z i8042 45056 0 2024-08-20T22:19:20.0401500Z serio 28672 3 i8042 2024-08-20T22:19:20.0401912Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:20.0402422Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:20.0402906Z button 24576 0 2024-08-20T22:19:20.0403271Z sunrpc 692224 1 2024-08-20T22:19:20.0403638Z sch_fq_codel 20480 17 2024-08-20T22:19:20.0404005Z dm_mod 188416 0 2024-08-20T22:19:20.0404362Z fuse 163840 1 2024-08-20T22:19:20.0404724Z dax 45056 1 dm_mod 2024-08-20T22:19:20.0405117Z configfs 57344 1 2024-08-20T22:19:20.0405477Z loop 36864 0 2024-08-20T22:19:20.0405877Z dmi_sysfs 20480 0 2024-08-20T22:19:20.0406406Z crc32_pclmul 16384 0 2024-08-20T22:19:20.0406898Z crc32c_intel 24576 0 2024-08-20T22:19:20.0407376Z efivarfs 24576 1 2024-08-20T22:19:20.0407855Z + modinfo nvidia 2024-08-20T22:19:20.0408573Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:20.0409270Z alias: char-major-195-* 2024-08-20T22:19:20.0409650Z version: 550.54.15 2024-08-20T22:19:20.0409993Z supported: external 2024-08-20T22:19:20.0410326Z license: NVIDIA 2024-08-20T22:19:20.0410702Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:20.0411179Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:20.0411628Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:20.0412090Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:20.0412584Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:20.0413071Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:20.0413550Z depends: i2c-core,drm 2024-08-20T22:19:20.0413912Z retpoline: Y 2024-08-20T22:19:20.0414221Z name: nvidia 2024-08-20T22:19:20.0414960Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:20.0415721Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:20.0416396Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:20.0416987Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:20.0417426Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:20.0417855Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:20.0418310Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:20.0418740Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:20.0419166Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:20.0419698Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:20.0420250Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:20.0420723Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:20.0421142Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:20.0421658Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:20.0422111Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:20.0422634Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:20.0423210Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:20.0423761Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:20.0424488Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:20.0425080Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:20.0425679Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:20.0426286Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:20.0426908Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:20.0427474Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:20.0428002Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:20.0428492Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:20.0428962Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:20.0429441Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:20.0429902Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:20.0430342Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:20.0430838Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:20.0431355Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:20.0431820Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:20.0432299Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:20.0432792Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:20.0433263Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:20.0433756Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:20.0434227Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:20.0434636Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:20.0435091Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:20.0435565Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:20.0436016Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:20.0436490Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:20.0436951Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:20.0437403Z parm: rm_firmware_active:charp 2024-08-20T22:19:20.0437827Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:20.0438225Z ++ command -v nvidia-smi 2024-08-20T22:19:20.0438623Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:20.0438991Z + set +e 2024-08-20T22:19:20.0439503Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:22.2832685Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:22.2833285Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:22.2833986Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:22.2834415Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:22.2834797Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:22.2835833Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:22.2836576Z + set -e 2024-08-20T22:19:22.2836890Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:22.2837461Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:22.2838145Z + post_install_nvidia_driver_common 2024-08-20T22:19:22.2838738Z + sudo modprobe nvidia 2024-08-20T22:19:22.3973403Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:22.3973842Z + lspci 2024-08-20T22:19:22.3974136Z After installing NVIDIA driver 2024-08-20T22:19:22.4081857Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:22.4082667Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:22.4083613Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:22.4084411Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:22.4085249Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:22.4086298Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:22.4087027Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:22.4087836Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:22.4088425Z + lsmod 2024-08-20T22:19:22.4115173Z Module Size Used by 2024-08-20T22:19:22.4115586Z nvidia_uvm 4706304 0 2024-08-20T22:19:22.4116082Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:22.4116661Z drm 602112 1 nvidia 2024-08-20T22:19:22.4117238Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:22.4117708Z backlight 24576 1 drm 2024-08-20T22:19:22.4118129Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:22.4118553Z xt_conntrack 16384 1 2024-08-20T22:19:22.4118978Z nft_chain_nat 16384 3 2024-08-20T22:19:22.4119494Z xt_MASQUERADE 20480 1 2024-08-20T22:19:22.4120106Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:22.4120626Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:22.4121203Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:22.4121847Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:22.4122319Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:22.4122749Z xfrm_user 57344 1 2024-08-20T22:19:22.4123152Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:22.4123579Z xt_addrtype 16384 2 2024-08-20T22:19:22.4123960Z nft_compat 20480 4 2024-08-20T22:19:22.4124413Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:22.4125030Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:22.4125575Z br_netfilter 36864 0 2024-08-20T22:19:22.4125991Z bridge 307200 1 br_netfilter 2024-08-20T22:19:22.4126441Z stp 16384 1 bridge 2024-08-20T22:19:22.4126870Z llc 16384 2 bridge,stp 2024-08-20T22:19:22.4127294Z overlay 167936 0 2024-08-20T22:19:22.4127671Z tls 114688 0 2024-08-20T22:19:22.4128038Z nls_ascii 16384 1 2024-08-20T22:19:22.4128413Z nls_cp437 20480 1 2024-08-20T22:19:22.4128788Z vfat 24576 1 2024-08-20T22:19:22.4129156Z fat 86016 1 vfat 2024-08-20T22:19:22.4129554Z ena 167936 0 2024-08-20T22:19:22.4129929Z ptp 36864 1 ena 2024-08-20T22:19:22.4130344Z pps_core 24576 1 ptp 2024-08-20T22:19:22.4130752Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:22.4131136Z aesni_intel 393216 0 2024-08-20T22:19:22.4131504Z i8042 45056 0 2024-08-20T22:19:22.4131879Z serio 28672 3 i8042 2024-08-20T22:19:22.4132298Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:22.4133037Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:22.4133541Z button 24576 0 2024-08-20T22:19:22.4133908Z sunrpc 692224 1 2024-08-20T22:19:22.4134314Z sch_fq_codel 20480 17 2024-08-20T22:19:22.4134821Z dm_mod 188416 0 2024-08-20T22:19:22.4135304Z fuse 163840 1 2024-08-20T22:19:22.4135790Z dax 45056 1 dm_mod 2024-08-20T22:19:22.4136304Z configfs 57344 1 2024-08-20T22:19:22.4136752Z loop 36864 0 2024-08-20T22:19:22.4137119Z dmi_sysfs 20480 0 2024-08-20T22:19:22.4137484Z crc32_pclmul 16384 0 2024-08-20T22:19:22.4137855Z crc32c_intel 24576 0 2024-08-20T22:19:22.4138221Z efivarfs 24576 1 2024-08-20T22:19:22.4138584Z + modinfo nvidia 2024-08-20T22:19:22.4139290Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:22.4140133Z alias: char-major-195-* 2024-08-20T22:19:22.4140520Z version: 550.54.15 2024-08-20T22:19:22.4140867Z supported: external 2024-08-20T22:19:22.4141204Z license: NVIDIA 2024-08-20T22:19:22.4141585Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:22.4142068Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:22.4142535Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:22.4143002Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:22.4143499Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:22.4143982Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:22.4144628Z depends: i2c-core,drm 2024-08-20T22:19:22.4144984Z retpoline: Y 2024-08-20T22:19:22.4145300Z name: nvidia 2024-08-20T22:19:22.4145964Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:22.4146665Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:22.4147305Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:22.4147898Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:22.4148339Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:22.4148771Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:22.4149218Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:22.4149654Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:22.4150093Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:22.4150607Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:22.4151162Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:22.4151639Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:22.4152066Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:22.4152477Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:22.4152940Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:22.4153465Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:22.4154056Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:22.4154603Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:22.4155207Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:22.4155824Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:22.4156452Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:22.4157030Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:22.4157522Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:22.4158060Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:22.4158600Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:22.4159084Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:22.4159548Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:22.4160026Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:22.4160504Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:22.4161047Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:22.4161554Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:22.4162076Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:22.4162552Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:22.4163032Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:22.4163529Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:22.4164013Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:22.4164508Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:22.4164991Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:22.4165401Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:22.4165865Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:22.4166324Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:22.4166786Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:22.4167413Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:22.4167887Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:22.4168331Z parm: rm_firmware_active:charp 2024-08-20T22:19:22.4168735Z + set +e 2024-08-20T22:19:22.4169052Z + nvidia-smi 2024-08-20T22:19:23.9862011Z Tue Aug 20 22:19:23 2024 2024-08-20T22:19:23.9863007Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:23.9863950Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:23.9864884Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:23.9865697Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:23.9866587Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:23.9867275Z | | | MIG M. | 2024-08-20T22:19:23.9867857Z |=========================================+========================+======================| 2024-08-20T22:19:23.9960844Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:23.9961695Z | 0% 29C P0 56W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:23.9962338Z | | | N/A | 2024-08-20T22:19:23.9963064Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:23.9963734Z 2024-08-20T22:19:23.9964407Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:23.9965030Z | Processes: | 2024-08-20T22:19:23.9965749Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:23.9966446Z | ID ID Usage | 2024-08-20T22:19:23.9967010Z |=========================================================================================| 2024-08-20T22:19:23.9967656Z | No running processes found | 2024-08-20T22:19:23.9968428Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:24.6141381Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:26.1785182Z NVIDIA A10G 2024-08-20T22:19:26.6422162Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:26.6422788Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:26.6423547Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:26.6424093Z + set -e 2024-08-20T22:19:26.6424515Z INFO: Ignoring allowed status 0 2024-08-20T22:19:26.6431356Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:26.6435765Z + sudo yum install -y yum-utils 2024-08-20T22:19:27.1314578Z Last metadata expiration check: 0:09:37 ago on Tue Aug 20 22:09:50 2024. 2024-08-20T22:19:27.1536919Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:27.1839523Z Dependencies resolved. 2024-08-20T22:19:27.1964740Z Nothing to do. 2024-08-20T22:19:27.1965104Z Complete! 2024-08-20T22:19:27.2918683Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:27.2920178Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:27.2922048Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:27.5581654Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:27.6325877Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:28.1383296Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:19:28.1601741Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:28.1895981Z Dependencies resolved. 2024-08-20T22:19:28.2020261Z Nothing to do. 2024-08-20T22:19:28.2020982Z Complete! 2024-08-20T22:19:28.3016426Z + sudo systemctl restart docker 2024-08-20T22:20:24.2117303Z Tue Aug 20 22:20:24 2024 2024-08-20T22:20:24.2118329Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:24.2119325Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:24.2120291Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:24.2121266Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:24.2122354Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:24.2123220Z | | | MIG M. | 2024-08-20T22:20:24.2123865Z |=========================================+========================+======================| 2024-08-20T22:20:24.2245816Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:24.2246709Z | 0% 30C P0 56W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:24.2247475Z | | | N/A | 2024-08-20T22:20:24.2248384Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:24.2249103Z 2024-08-20T22:20:24.2249910Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:24.2250704Z | Processes: | 2024-08-20T22:20:24.2251577Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:24.2252404Z | ID ID Usage | 2024-08-20T22:20:24.2253106Z |=========================================================================================| 2024-08-20T22:20:24.2253923Z | No running processes found | 2024-08-20T22:20:24.2254883Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:24.4344125Z Command completed after 1 attempt(s). 2024-08-20T22:20:24.4426686Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:24.4427455Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:24.4428139Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:24.4428806Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:24.4443208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:24.4443699Z env: 2024-08-20T22:20:24.4443974Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:24.4444424Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:24.4444897Z ##[endgroup] 2024-08-20T22:20:24.7539099Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:25.1734252Z Collecting psutil==5.9.1 2024-08-20T22:20:25.2111935Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:25.2701464Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:25.2749083Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:25.3663444Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:25.5249078Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:25.7245202Z Prepare all required actions 2024-08-20T22:20:25.7245693Z Getting action download info 2024-08-20T22:20:25.9027453Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:26.1132592Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:26.2314371Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:26.2314854Z with: 2024-08-20T22:20:26.2315183Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:26.2315640Z s3-bucket: gha-artifacts 2024-08-20T22:20:26.2315975Z env: 2024-08-20T22:20:26.2316243Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:26.2316679Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:26.2317158Z ##[endgroup] 2024-08-20T22:20:26.2350950Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:26.2351399Z with: 2024-08-20T22:20:26.2351776Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:26.2352226Z s3-bucket: gha-artifacts 2024-08-20T22:20:26.2352587Z region: us-east-1 2024-08-20T22:20:26.2352884Z env: 2024-08-20T22:20:26.2353156Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:26.2353594Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:26.2354070Z ##[endgroup] 2024-08-20T22:20:26.7223731Z (node:52535) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:26.7224876Z 2024-08-20T22:20:26.7225143Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:26.7225873Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:26.7226789Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:26.7921813Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:26.7922938Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:34.3431794Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:34.3438632Z Artifact download has finished successfully 2024-08-20T22:20:34.3796337Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:34.3796780Z unzip -o artifacts.zip 2024-08-20T22:20:34.3805652Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:34.3806154Z env: 2024-08-20T22:20:34.3806427Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:34.3806907Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:34.3807413Z ##[endgroup] 2024-08-20T22:20:34.4078114Z Archive: artifacts.zip 2024-08-20T22:20:34.4079273Z creating: dist/ 2024-08-20T22:20:36.5373515Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:36.5374372Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:36.5375013Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:36.5375764Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:36.5376595Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.5382244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.5383429Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.5384630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.5385624Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.5386594Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.5388526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.5390078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.5391287Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.5392303Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.5393488Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.5394943Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.5396470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.5397617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.5399198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.5400599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.5401653Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.5402660Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.5442449Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.5482629Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.5484073Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.5530529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.5533355Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.5536225Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.5538314Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.5539756Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.5541149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.5542541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.5543924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.5545440Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.5546720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.5548013Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.5549241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.5550467Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.5551662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.5552876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.5605333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.5665302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.5666489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.5667497Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.5668428Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.5669338Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.5670261Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:36.5671431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:36.5672746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:36.5673861Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:36.5674899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:36.5675989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:36.5677073Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:36.5678190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:36.5679285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:36.5680353Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:36.5697808Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:36.5841169Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:36.5842184Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:36.5843279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:36.5844492Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:36.5845648Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:36.5846739Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:36.5847868Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:36.5849009Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:36.5850151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:36.5851294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:36.5852410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:36.5870569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:36.5955024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:36.5957400Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.5958690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.5959834Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.5960870Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.5961770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.5962668Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:36.5963532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:36.5964317Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:36.5965131Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:36.6084678Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:36.6147622Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:36.6148376Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:36.6149086Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:36.6149905Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.6156486Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.6157425Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.6158354Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.6159342Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.6160306Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.6161392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.6162629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.6163645Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.6164631Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.6166255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.6167947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.6170100Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.6171243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.6172845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.6174319Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.6175364Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.6176354Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.6216125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.6257454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.6259166Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.6303743Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.6305283Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.6307040Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.6308503Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.6310213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.6311684Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.6313066Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.6314433Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.6315790Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.6317082Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.6318314Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.6319517Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.6320778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.6322016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.6323218Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.6378727Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.6438881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.6440060Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.6441042Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.6441901Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.6442790Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.6443726Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:36.6444800Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:36.6445990Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:36.6447141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:36.6448224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:36.6449330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:36.6450438Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:36.6451552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:36.6452654Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:36.6453749Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:36.6470814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:36.6535295Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:36.6536471Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.6537533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.6538511Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.6539408Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.6540302Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.6541196Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:36.6543623Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:36.6545523Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:36.6547280Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:36.6595759Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:36.6596621Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:36.6597381Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:36.6598268Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.6604458Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.6605605Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.6606676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.6607761Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.6608885Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.6610059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.6611412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.6612481Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.6613538Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.6614817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.6616364Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.6617982Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.6619407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.6621311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.6622879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.6624058Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.6625230Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.6664716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.6704730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.6706666Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.6752406Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.6755352Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.6757862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.6759408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.6760904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.6762381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.6763856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.6765305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.6766723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.6768118Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.6769421Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.6770709Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.6772193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.6773460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.6774729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.6827542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.6886943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.6888192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.6889244Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.6890162Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.6891142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.6892153Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:36.6893289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:36.6894541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:36.6895743Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:36.6896880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:36.6898062Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:36.6899412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:36.6900723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:36.6901897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:36.6903067Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:36.6904309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:36.7029744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:36.7032055Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:36.7034405Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:36.7037020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:36.7038521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:36.7039713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:36.7040939Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:36.7042157Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:36.7043386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:36.7044618Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:36.7045848Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:36.7058026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:36.7113540Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:36.7114834Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.7115971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.7117008Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.7117965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.7118919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.7119875Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:36.7121794Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:36.7122746Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:36.7123713Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:36.7227547Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:36.7270028Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:36.7270730Z creating: build/lib/ 2024-08-20T22:20:36.7359052Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:36.7814259Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:36.7824094Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:36.7832192Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:36.7840332Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:36.7841008Z inflating: build/lib/libclog.a 2024-08-20T22:20:36.7859508Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:36.7862363Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:36.7927409Z inflating: build/lib/libgtest.a 2024-08-20T22:20:36.8005302Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:36.8068486Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:36.8076400Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:36.8104980Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:36.8232224Z inflating: build/lib/libgloo.a 2024-08-20T22:20:36.8253352Z inflating: build/lib/libfmt.a 2024-08-20T22:20:36.8348491Z inflating: build/lib/libc10.so 2024-08-20T22:20:36.8350198Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:36.8351760Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:36.8371129Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:36.8874425Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:36.8892477Z inflating: build/lib/libgmock.a 2024-08-20T22:20:36.8893070Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:36.8893940Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:36.9283941Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:37.9462838Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:38.0036268Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:38.0093636Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:38.0094205Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:38.1353183Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:38.1608818Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:38.2114918Z inflating: build/lib/libkineto.a 2024-08-20T22:20:38.2301823Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:38.2344360Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:38.3037574Z inflating: build/lib/libonnx.a 2024-08-20T22:20:40.7499479Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:40.7504207Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:40.7509408Z inflating: build/lib/libshm.so 2024-08-20T22:20:42.8149165Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:42.8150802Z inflating: build/lib/libtorch.so 2024-08-20T22:20:42.8153741Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:43.6325106Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:43.8321016Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:43.8393352Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:43.8414171Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:43.8440591Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:43.8466016Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:43.8501036Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:43.8501671Z creating: build/bin/ 2024-08-20T22:20:43.8552040Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:43.8603562Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:43.8655784Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:43.8714720Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:43.8768163Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:43.8817621Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:43.8868386Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:43.8923587Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:43.8979510Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:43.9035786Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:43.9107262Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:43.9160214Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:43.9209435Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:43.9259509Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:43.9310621Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:43.9366940Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:43.9422437Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:43.9472777Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:43.9529060Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:43.9580764Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:43.9632610Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:43.9682276Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:43.9735036Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:43.9790421Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:43.9841131Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:43.9898318Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:43.9953837Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:44.0007083Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:44.0057597Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:44.0107680Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:44.0273274Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:44.0324195Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:44.0378557Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:44.0435474Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:44.0498496Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:44.0574947Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:44.0727321Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:44.0781042Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:44.0832880Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:44.0885325Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:44.0935821Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:44.0995132Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:44.1043601Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:44.1100101Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:44.1548765Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:44.1996595Z inflating: build/bin/protoc 2024-08-20T22:20:44.2049923Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:44.2103205Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:44.2156301Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:44.2208665Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:44.2261757Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:44.2315235Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:44.2364318Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:44.2417562Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:44.2748555Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:44.3095658Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:44.3455973Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:44.3509185Z inflating: build/bin/BackoffTest 2024-08-20T22:20:44.3561728Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:44.3617705Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:44.3671309Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:44.3686074Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:44.3740201Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:44.3744344Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:44.3747520Z inflating: build/bin/example_allreduce 2024-08-20T22:20:44.3803206Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:44.3872423Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:44.3875295Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:44.3943029Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:44.3953996Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:44.4308377Z inflating: build/bin/test_lazy 2024-08-20T22:20:44.5492794Z inflating: build/bin/test_api 2024-08-20T22:20:44.5566411Z inflating: build/bin/Dict_test 2024-08-20T22:20:44.5619543Z inflating: build/bin/Dimname_test 2024-08-20T22:20:44.5685818Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:44.5743311Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:44.5802929Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:44.5862256Z inflating: build/bin/atest 2024-08-20T22:20:44.5925708Z inflating: build/bin/basic 2024-08-20T22:20:44.5981230Z inflating: build/bin/broadcast_test 2024-08-20T22:20:44.6032412Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:44.6091147Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:44.6144207Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:44.6239167Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:44.6289561Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:44.6340761Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:44.6399335Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:44.6454793Z inflating: build/bin/half_test 2024-08-20T22:20:44.6551120Z inflating: build/bin/ivalue_test 2024-08-20T22:20:44.6601197Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:44.6655986Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:44.6710440Z inflating: build/bin/memory_format_test 2024-08-20T22:20:44.6764095Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:44.6818132Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:44.6875180Z inflating: build/bin/native_test 2024-08-20T22:20:44.6926164Z inflating: build/bin/operator_name_test 2024-08-20T22:20:44.6978364Z inflating: build/bin/operators_test 2024-08-20T22:20:44.7030974Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:44.7098670Z inflating: build/bin/pow_test 2024-08-20T22:20:44.7158470Z inflating: build/bin/quantized_test 2024-08-20T22:20:44.7208952Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:44.7260549Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:44.7317419Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:44.7376902Z inflating: build/bin/scalar_test 2024-08-20T22:20:44.7428579Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:44.7481767Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:44.7561481Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:44.7615727Z inflating: build/bin/test_parallel 2024-08-20T22:20:44.7618820Z inflating: build/bin/thread_init_test 2024-08-20T22:20:44.7674915Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:44.7735270Z inflating: build/bin/type_test 2024-08-20T22:20:44.7787871Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:44.7789608Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:44.7859061Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:44.7911457Z inflating: build/bin/weakref_test 2024-08-20T22:20:44.7962764Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:44.8014800Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:44.8075118Z inflating: build/bin/IListRef_test 2024-08-20T22:20:44.8181814Z inflating: build/bin/List_test 2024-08-20T22:20:44.8248742Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:44.8371547Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:44.8469067Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:44.8597967Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:44.8701970Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:44.8763378Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:44.8861132Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:44.8912068Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:44.8968257Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:44.9017577Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:44.9322357Z inflating: build/bin/op_registration_test 2024-08-20T22:20:44.9386257Z inflating: build/bin/inline_container_test 2024-08-20T22:20:44.9439496Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:44.9492493Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:44.9551466Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:44.9606357Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:44.9676601Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:44.9736026Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:44.9785793Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:44.9844465Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:44.9895731Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:44.9960750Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:45.0017664Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:45.0067862Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:45.0119017Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:45.0168446Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:45.0222212Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:45.0274796Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:45.0325013Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:45.0386013Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:45.0435427Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:45.0488011Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:45.0502458Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:45.0570201Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:45.0627833Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:45.0691818Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:45.0754322Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:45.1584623Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:45.2168046Z inflating: build/bin/test_jit 2024-08-20T22:20:45.2169217Z creating: .additional_ci_files/ 2024-08-20T22:20:45.2230223Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:45.2470526Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:45.2511515Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:45.2511910Z rm artifacts.zip 2024-08-20T22:20:45.2522732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:45.2523238Z env: 2024-08-20T22:20:45.2523507Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.2523956Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.2524436Z ##[endgroup] 2024-08-20T22:20:45.3632325Z ##[group]Run df -H 2024-08-20T22:20:45.3632643Z df -H 2024-08-20T22:20:45.3642114Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:45.3642603Z env: 2024-08-20T22:20:45.3642879Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.3643321Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.3643792Z ##[endgroup] 2024-08-20T22:20:45.3694763Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:45.3695485Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:45.3696108Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:45.3696635Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:45.3697089Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:45.3697910Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:45.3698402Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:45.3698939Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:45.3740225Z Prepare all required actions 2024-08-20T22:20:45.3740660Z Getting action download info 2024-08-20T22:20:45.4784925Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:45.4785381Z with: 2024-08-20T22:20:45.4785635Z env: 2024-08-20T22:20:45.4785906Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.4786346Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.4786827Z ##[endgroup] 2024-08-20T22:20:45.4860222Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:45.4860664Z with: 2024-08-20T22:20:45.4860924Z name: td_results 2024-08-20T22:20:45.4861238Z s3-bucket: gha-artifacts 2024-08-20T22:20:45.4861587Z region: us-east-1 2024-08-20T22:20:45.4861880Z env: 2024-08-20T22:20:45.4862154Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.4862602Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.4863082Z ##[endgroup] 2024-08-20T22:20:45.9725941Z (node:52557) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:45.9726664Z 2024-08-20T22:20:45.9726932Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:45.9727648Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:45.9728516Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:46.0484129Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:46.0490732Z Artifact download has finished successfully 2024-08-20T22:20:46.0810220Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:46.0810703Z mkdir -p .additional_ci_files 2024-08-20T22:20:46.0811258Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:46.0820993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:46.0821479Z env: 2024-08-20T22:20:46.0821755Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.0822216Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.0822696Z ##[endgroup] 2024-08-20T22:20:46.0882566Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:46.0897176Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:46.0932146Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:46.0932637Z .github/scripts/parse_ref.py 2024-08-20T22:20:46.0941322Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:46.0941670Z env: 2024-08-20T22:20:46.0941944Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.0942389Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.0942860Z ##[endgroup] 2024-08-20T22:20:46.1229463Z Prepare all required actions 2024-08-20T22:20:46.1267882Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:46.1268344Z with: 2024-08-20T22:20:46.1268882Z github-token: *** 2024-08-20T22:20:46.1269185Z env: 2024-08-20T22:20:46.1269450Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.1269901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.1270404Z ##[endgroup] 2024-08-20T22:20:46.1287524Z ##[group]Run set -eux 2024-08-20T22:20:46.1287859Z set -eux 2024-08-20T22:20:46.1288441Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:46.1297056Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:46.1297542Z env: 2024-08-20T22:20:46.1297817Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.1298265Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.1298897Z GITHUB_TOKEN: *** 2024-08-20T22:20:46.1299197Z ##[endgroup] 2024-08-20T22:20:46.1327965Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-024c42853ba21bbc0 2024-08-20T22:20:47.6341057Z setting job-id=29026529670 2024-08-20T22:20:47.6342016Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.6557826Z Prepare all required actions 2024-08-20T22:20:47.6558263Z Getting action download info 2024-08-20T22:20:47.7713406Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:47.7713859Z with: 2024-08-20T22:20:47.7714305Z github-token: *** 2024-08-20T22:20:47.7723145Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:47.7732634Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.7733413Z env: 2024-08-20T22:20:47.7733689Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.7734133Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.7734609Z ##[endgroup] 2024-08-20T22:20:47.7776963Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:47.7777513Z with: 2024-08-20T22:20:47.7777778Z shell: bash 2024-08-20T22:20:47.7778076Z timeout_minutes: 10 2024-08-20T22:20:47.7778393Z max_attempts: 5 2024-08-20T22:20:47.7778703Z retry_wait_seconds: 30 2024-08-20T22:20:47.7779808Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:47.7780972Z polling_interval_seconds: 1 2024-08-20T22:20:47.7781343Z warning_on_retry: true 2024-08-20T22:20:47.7781679Z continue_on_error: false 2024-08-20T22:20:47.7782012Z env: 2024-08-20T22:20:47.7782279Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.7782712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.7783345Z GITHUB_TOKEN: *** 2024-08-20T22:20:47.7783653Z ##[endgroup] 2024-08-20T22:20:47.8544284Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:48.0922929Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:48.2391497Z Collecting requests==2.27.1 2024-08-20T22:20:48.2751163Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:48.4664201Z Collecting pyyaml==6.0.1 2024-08-20T22:20:48.4705604Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:48.4997182Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:48.5454661Z Collecting certifi>=2017.4.17 2024-08-20T22:20:48.5495978Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:48.5582572Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:48.8572778Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:48.8615755Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:48.9419888Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:49.0663206Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:49.8338743Z Command completed after 1 attempt(s). 2024-08-20T22:20:49.8397615Z ##[group]Run set -x 2024-08-20T22:20:49.8397951Z set -x 2024-08-20T22:20:49.8398245Z  2024-08-20T22:20:49.8398785Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:49.8399452Z # in runner workspace 2024-08-20T22:20:49.8399971Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:49.8409420Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.8409909Z env: 2024-08-20T22:20:49.8410179Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.8410617Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.8411105Z ##[endgroup] 2024-08-20T22:20:49.8441512Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:49.8681227Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:49.8681811Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:49.8682273Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:49.8682661Z  2024-08-20T22:20:49.8683200Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:49.8683872Z # in runner workspace 2024-08-20T22:20:49.8684444Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:49.8685071Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:49.8685528Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:49.8694765Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:49.8704212Z  --selected-test-configs "" \ 2024-08-20T22:20:49.8704786Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:49.8705205Z  --tag "${TAG}" \ 2024-08-20T22:20:49.8705595Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:49.8706033Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:49.8706457Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:49.8714591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.8715087Z env: 2024-08-20T22:20:49.8715355Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.8715803Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.8716462Z GITHUB_TOKEN: *** 2024-08-20T22:20:49.8717174Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:49.8717954Z PR_NUMBER: 2024-08-20T22:20:49.8718261Z TAG: ciflow/inductor/133712 2024-08-20T22:20:49.8718634Z EVENT_NAME: push 2024-08-20T22:20:49.8718941Z SCHEDULE: 2024-08-20T22:20:49.8719220Z HEAD_BRANCH: 2024-08-20T22:20:49.8719523Z ##[endgroup] 2024-08-20T22:20:49.8750449Z Workflow: inductor 2024-08-20T22:20:49.8751495Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:50.1091012Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:50.2411492Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:50.2411951Z echo "Filtered matrix:" 2024-08-20T22:20:50.2421016Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:50.2430476Z  2024-08-20T22:20:50.2430751Z echo 2024-08-20T22:20:50.2431125Z echo "Is the current job unstable? False" 2024-08-20T22:20:50.2431573Z  2024-08-20T22:20:50.2431851Z echo 2024-08-20T22:20:50.2432204Z echo "Is keep-going label set? False" 2024-08-20T22:20:50.2432642Z  2024-08-20T22:20:50.2432909Z echo 2024-08-20T22:20:50.2433219Z echo "Renabled issues? " 2024-08-20T22:20:50.2442000Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:50.2442540Z env: 2024-08-20T22:20:50.2442815Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.2443260Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.2443740Z ##[endgroup] 2024-08-20T22:20:50.2471277Z Filtered matrix: 2024-08-20T22:20:50.2482159Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:50.2490972Z 2024-08-20T22:20:50.2491119Z Is the current job unstable? False 2024-08-20T22:20:50.2491399Z 2024-08-20T22:20:50.2491760Z Is keep-going label set? False 2024-08-20T22:20:50.2492024Z 2024-08-20T22:20:50.2492171Z Renabled issues? 2024-08-20T22:20:50.2535553Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:50.2536263Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:50.2544796Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:50.2545287Z env: 2024-08-20T22:20:50.2545578Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.2546028Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.2546509Z JOB_TIMEOUT: 240 2024-08-20T22:20:50.2546805Z ##[endgroup] 2024-08-20T22:20:50.2633773Z ##[group]Run set -x 2024-08-20T22:20:50.2634167Z set -x 2024-08-20T22:20:50.2634463Z  2024-08-20T22:20:50.2634813Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:50.2635347Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:50.2635903Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:50.2636403Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:50.2636811Z else 2024-08-20T22:20:50.2637145Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:50.2637559Z fi 2024-08-20T22:20:50.2637829Z  2024-08-20T22:20:50.2638307Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:50.2639050Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:50.2639719Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:50.2640300Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:50.2640751Z container_name=$(docker run \ 2024-08-20T22:20:50.2641163Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:50.2641535Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:50.2641933Z  -e PR_NUMBER \ 2024-08-20T22:20:50.2642286Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:50.2642673Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:50.2643074Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:50.2643454Z  -e GITHUB_JOB \ 2024-08-20T22:20:50.2643804Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:50.2644190Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:50.2644590Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:50.2644986Z  -e JOB_ID \ 2024-08-20T22:20:50.2645317Z  -e JOB_NAME \ 2024-08-20T22:20:50.2645660Z  -e BASE_SHA \ 2024-08-20T22:20:50.2646001Z  -e BRANCH \ 2024-08-20T22:20:50.2646324Z  -e SHA1 \ 2024-08-20T22:20:50.2646669Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:50.2647072Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:50.2647444Z  -e SHARD_NUMBER \ 2024-08-20T22:20:50.2647804Z  -e TEST_CONFIG \ 2024-08-20T22:20:50.2648175Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:50.2648562Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:50.2648968Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:50.2649382Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:50.2649777Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:50.2650165Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:50.2650535Z  -e NO_TD \ 2024-08-20T22:20:50.2650867Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:50.2651251Z  -e PR_LABELS \ 2024-08-20T22:20:50.2651657Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:50.2652100Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:50.2652494Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:50.2652901Z  -e XLA_CUDA \ 2024-08-20T22:20:50.2653303Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:50.2653801Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:50.2654318Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:50.2654826Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:50.2655287Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:50.2655729Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:50.2656161Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:50.2656634Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:50.2657181Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:50.2657809Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:50.2658193Z  --ipc=host \ 2024-08-20T22:20:50.2658547Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:50.2658939Z  --tty \ 2024-08-20T22:20:50.2659245Z  --detach \ 2024-08-20T22:20:50.2659722Z  --name="${container_name}" \ 2024-08-20T22:20:50.2660138Z  --user jenkins \ 2024-08-20T22:20:50.2660616Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:50.2661161Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:50.2661594Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:50.2661987Z ) 2024-08-20T22:20:50.2662390Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:50.2663296Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:50.2664248Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:50.2665253Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:50.2674750Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:50.2675094Z env: 2024-08-20T22:20:50.2675369Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.2675825Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.2676421Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:50.2676909Z PR_NUMBER: 2024-08-20T22:20:50.2677231Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:50.2677641Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:50.2677988Z GITHUB_JOB: test 2024-08-20T22:20:50.2678294Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:50.2678654Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:50.2679004Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:50.2679334Z JOB_ID: 29026529670 2024-08-20T22:20:50.2680046Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:50.2680830Z BRANCH: 2024-08-20T22:20:50.2681164Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:50.2681683Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:50.2682172Z TEST_CONFIG: dynamic_inductor_torchbench 2024-08-20T22:20:50.2682601Z SHARD_NUMBER: 2 2024-08-20T22:20:50.2682904Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:50.2683221Z REENABLED_ISSUES: 2024-08-20T22:20:50.2683549Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:50.2683929Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:50.2684279Z TEST_SHOWLOCALS: False 2024-08-20T22:20:50.2684616Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:50.2684943Z NO_TD: False 2024-08-20T22:20:50.2685240Z TD_DISTRIBUTED: False 2024-08-20T22:20:50.2685648Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:50.2686126Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:50.2686498Z SHM_SIZE: 2g 2024-08-20T22:20:50.2687576Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:50.2688739Z XLA_CUDA: 2024-08-20T22:20:50.2689206Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:50.2689808Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:50.2690240Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:50.2690642Z DASHBOARD_TAG: 2024-08-20T22:20:50.2691147Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:50.2691516Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:50.2691884Z ##[endgroup] 2024-08-20T22:20:50.2721795Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:50.2722606Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:50.2723111Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:50.2730947Z +++ nproc --ignore=2 2024-08-20T22:20:50.2761516Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:21:01.5486369Z + container_name=1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-20T22:21:01.5491475Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:21:01.5493961Z + docker exec -i 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:21:01.7141284Z + echo DOCKER_CONTAINER_ID=1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-20T22:21:01.7145112Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:21:01.7148085Z + docker exec -t 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:21:02.1111660Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:21:02.4375399Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:21:02.4377590Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:21:02.4379589Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:21:02.4381776Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:21:02.4384620Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:21:02.4390149Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:21:02.4406199Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:21:02.4419399Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:21:02.4436811Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:21:02.4825692Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:21:03.0626673Z Installing collected packages: torch 2024-08-20T22:21:13.0230875Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:13.0232651Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:13.0233377Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:13.1045189Z + export TERM=vt100 2024-08-20T22:21:13.1045574Z + TERM=vt100 2024-08-20T22:21:13.1049517Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:13.1059131Z + source .ci/pytorch/common.sh 2024-08-20T22:21:13.1062392Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:13.1072185Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:13.1073668Z +++ declare -f -t trap_add 2024-08-20T22:21:13.1079568Z ++ set -ex 2024-08-20T22:21:13.1080228Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:13.1080884Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:13.1081424Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:13.1083860Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:13.1103191Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:13.1103776Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:13.1104177Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:13.1104674Z + shift 2024-08-20T22:21:13.1106771Z + for trap_add_name in "$@" 2024-08-20T22:21:13.1110814Z +++ trap -p EXIT 2024-08-20T22:21:13.1114027Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:13.1114493Z +++ extract_trap_cmd 2024-08-20T22:21:13.1114888Z +++ printf '%s\n' '' 2024-08-20T22:21:13.1115294Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:13.1117211Z + trap -- ' 2024-08-20T22:21:13.1117630Z cleanup_workspace' EXIT 2024-08-20T22:21:13.1118108Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:13.8202870Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:13.8224279Z + echo 'Environment variables:' 2024-08-20T22:21:13.8224847Z Environment variables: 2024-08-20T22:21:13.8225280Z + env 2024-08-20T22:21:13.8234698Z INSTALLED_DB=yes 2024-08-20T22:21:13.8235243Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:13.8235783Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:13.8236259Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.8237012Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:13.8237916Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:13.8238638Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:13.8239264Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.8239845Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:13.8240343Z HOSTNAME=1b6b87892a85 2024-08-20T22:21:13.8241192Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8242053Z GITHUB_ACTION=__self 2024-08-20T22:21:13.8242553Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:13.8247327Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:13.8251355Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:13.8251863Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:13.8252214Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:13.8252588Z TEST_CONFIG=dynamic_inductor_torchbench 2024-08-20T22:21:13.8253290Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:13.8253764Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:13.8254225Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.8254664Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:13.8255221Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:13.8255638Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:13.8256079Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.8256568Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:13.8257021Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:13.8257343Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:13.8257723Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:13.8258107Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8258547Z XLA_CUDA= 2024-08-20T22:21:13.8259033Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:13.8261198Z *** 2024-08-20T22:21:13.8261492Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:13.8261895Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:13.8262257Z GITHUB_ACTIONS=true 2024-08-20T22:21:13.8262587Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:13.8263094Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:13.8263629Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:13.8264102Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8264787Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:13.8265258Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8266042Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.8266745Z UCC_HOME=/usr 2024-08-20T22:21:13.8267083Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:13.8267473Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:13.8267811Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:13.8268278Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:13.8268775Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.8269245Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:13.8269619Z SHARD_NUMBER=2 2024-08-20T22:21:13.8269927Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:13.8270270Z HOME=/var/lib/jenkins 2024-08-20T22:21:13.8270633Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:13.8271066Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:13.8271543Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:13.8272362Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:13.8272733Z CUDA_VERSION=12.1.1 2024-08-20T22:21:13.8273166Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:13.8273597Z NUM_TEST_SHARDS=2 2024-08-20T22:21:13.8273888Z UCX_HOME=/usr 2024-08-20T22:21:13.8274404Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:13.8275485Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8276796Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:13.8278070Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8279174Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:13.8279822Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:13.8280147Z DASHBOARD_TAG= 2024-08-20T22:21:13.8280439Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:13.8280896Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:13.8281427Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:13.8282413Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8283307Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.8283709Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:13.8284083Z PR_NUMBER= 2024-08-20T22:21:13.8284361Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:13.8284865Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:13.8285300Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:13.8285759Z TERM=vt100 2024-08-20T22:21:13.8286086Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:13.8286491Z INSTALLED_VISION=yes 2024-08-20T22:21:13.8286800Z BRANCH= 2024-08-20T22:21:13.8287208Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:13.8287598Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:13.8288011Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:13.8288764Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:13.8289500Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:13.8289971Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:13.8290434Z REENABLED_ISSUES= 2024-08-20T22:21:13.8290725Z SHLVL=1 2024-08-20T22:21:13.8290978Z MAX_JOBS=14 2024-08-20T22:21:13.8291289Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:13.8291638Z NVARCH=x86_64 2024-08-20T22:21:13.8291938Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:13.8292377Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8292910Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:13.8293395Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:13.8294056Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:13.8294608Z GITHUB_JOB=test 2024-08-20T22:21:13.8295008Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:13.8295555Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:13.8296057Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:13.8296371Z TD_DISTRIBUTED=False 2024-08-20T22:21:13.8296743Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:13.8297173Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:13.8297583Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:13.8297942Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:13.8298285Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:13.8298630Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:13.8299700Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:13.8300751Z GITHUB_BASE_REF= 2024-08-20T22:21:13.8301070Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:13.8301432Z CI=true 2024-08-20T22:21:13.8301745Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.8302156Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:13.8302531Z JOB_ID=29026529670 2024-08-20T22:21:13.8302838Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:13.8303165Z GITHUB_HEAD_REF= 2024-08-20T22:21:13.8303468Z GITHUB_ACTION_REF= 2024-08-20T22:21:13.8303903Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:13.8304475Z TEST_SHOWLOCALS=False 2024-08-20T22:21:13.8304810Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:13.8305168Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:13.8306080Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8306890Z NO_TD=False 2024-08-20T22:21:13.8307189Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:13.8307550Z _=/usr/bin/env 2024-08-20T22:21:13.8308032Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:13.8466310Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:13.8467556Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:13.8468421Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:13.8469404Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:13.8470009Z + BUILD_DIR=build 2024-08-20T22:21:13.8470336Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:13.8470723Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:13.8471057Z + SHARD_NUMBER=2 2024-08-20T22:21:13.8471349Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:13.8471661Z + export VALGRIND=ON 2024-08-20T22:21:13.8472366Z + VALGRIND=ON 2024-08-20T22:21:13.8472843Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:13.8473331Z + [[ 0 == \1 ]] 2024-08-20T22:21:13.8473622Z + [[ False == \1 ]] 2024-08-20T22:21:13.8474093Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:13.8474770Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:13.8484987Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:13.8485914Z + [[ -n '' ]] 2024-08-20T22:21:13.8486304Z + echo 'Environment variables' 2024-08-20T22:21:13.8486675Z Environment variables 2024-08-20T22:21:13.8486988Z + env 2024-08-20T22:21:13.8495895Z INSTALLED_DB=yes 2024-08-20T22:21:13.8496718Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:13.8497313Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:13.8497848Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.8498645Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:13.8499384Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:13.8500077Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:13.8500762Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.8501430Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:13.8501916Z HOSTNAME=1b6b87892a85 2024-08-20T22:21:13.8502830Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8503631Z GITHUB_ACTION=__self 2024-08-20T22:21:13.8503974Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:13.8507910Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:13.8512124Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:13.8512633Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:13.8512984Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:13.8513345Z TEST_CONFIG=dynamic_inductor_torchbench 2024-08-20T22:21:13.8513782Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:13.8514253Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:13.8514723Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.8515161Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:13.8515643Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:13.8516043Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:13.8516490Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.8516985Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:13.8517443Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:13.8517764Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:13.8518133Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:13.8518525Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8518964Z XLA_CUDA= 2024-08-20T22:21:13.8519406Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:13.8519790Z *** 2024-08-20T22:21:13.8520062Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:13.8520452Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:13.8520811Z GITHUB_ACTIONS=true 2024-08-20T22:21:13.8521132Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:13.8521629Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:13.8522160Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:13.8522710Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8523222Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:13.8523912Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8524699Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.8525409Z UCC_HOME=/usr 2024-08-20T22:21:13.8525741Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:13.8526237Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:13.8526580Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:13.8527036Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:13.8527527Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.8527990Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:13.8528362Z SHARD_NUMBER=2 2024-08-20T22:21:13.8528662Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:13.8529016Z HOME=/var/lib/jenkins 2024-08-20T22:21:13.8529375Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:13.8529805Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:13.8530269Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:13.8530752Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:13.8531118Z CUDA_VERSION=12.1.1 2024-08-20T22:21:13.8531541Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:13.8531971Z NUM_TEST_SHARDS=2 2024-08-20T22:21:13.8532267Z UCX_HOME=/usr 2024-08-20T22:21:13.8532782Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:13.8533847Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8535155Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:13.8536519Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8537632Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:13.8538277Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:13.8538595Z DASHBOARD_TAG= 2024-08-20T22:21:13.8538910Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:13.8539412Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:13.8539942Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:13.8540952Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8541860Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.8542267Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:13.8542634Z PR_NUMBER= 2024-08-20T22:21:13.8542916Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:13.8543237Z VALGRIND=ON 2024-08-20T22:21:13.8543534Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:13.8543960Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:13.8544517Z TERM=vt100 2024-08-20T22:21:13.8544857Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:13.8545265Z INSTALLED_VISION=yes 2024-08-20T22:21:13.8545568Z BRANCH= 2024-08-20T22:21:13.8545864Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:13.8546263Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:13.8546676Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:13.8547424Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:13.8548159Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:13.8548640Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:13.8549102Z REENABLED_ISSUES= 2024-08-20T22:21:13.8549418Z SHLVL=1 2024-08-20T22:21:13.8549698Z MAX_JOBS=14 2024-08-20T22:21:13.8550013Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:13.8550353Z NVARCH=x86_64 2024-08-20T22:21:13.8550646Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:13.8551103Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.8551622Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:13.8552109Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:13.8552766Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:13.8553334Z GITHUB_JOB=test 2024-08-20T22:21:13.8553837Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:13.8554384Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:13.8554889Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:13.8555219Z TD_DISTRIBUTED=False 2024-08-20T22:21:13.8555587Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:13.8556116Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:13.8556535Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:13.8556894Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:13.8557234Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:13.8557584Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:13.8558613Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:13.8559662Z GITHUB_BASE_REF= 2024-08-20T22:21:13.8559976Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:13.8560347Z CI=true 2024-08-20T22:21:13.8560664Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.8561070Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:13.8561437Z JOB_ID=29026529670 2024-08-20T22:21:13.8561744Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:13.8562076Z GITHUB_HEAD_REF= 2024-08-20T22:21:13.8562518Z GITHUB_ACTION_REF= 2024-08-20T22:21:13.8562996Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:13.8563469Z TEST_SHOWLOCALS=False 2024-08-20T22:21:13.8563794Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:13.8564151Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:13.8565070Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_15ab97ce-3262-4ab1-80e7-1c0ffbfdc6d4 2024-08-20T22:21:13.8565881Z NO_TD=False 2024-08-20T22:21:13.8566173Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:13.8566534Z _=/usr/bin/env 2024-08-20T22:21:13.8566875Z + echo 'Testing pytorch' 2024-08-20T22:21:13.8567216Z Testing pytorch 2024-08-20T22:21:13.8567530Z + export LANG=C.UTF-8 2024-08-20T22:21:13.8567864Z + LANG=C.UTF-8 2024-08-20T22:21:13.8568153Z + PR_NUMBER= 2024-08-20T22:21:13.8568518Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:13.8569097Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:13.8569648Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-08-20T22:21:13.8570311Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:13.8570992Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:13.8571531Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:13.8572290Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:13.8572772Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-08-20T22:21:13.8573382Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:13.8574014Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:13.8574656Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:13.8575210Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:13.8575737Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:15.0531028Z Collecting ninja==1.10.2 2024-08-20T22:21:15.0696796Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:15.1242323Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:15.7260831Z Installing collected packages: ninja 2024-08-20T22:21:15.7340444Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:15.7341721Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:15.7897794Z Successfully installed ninja-1.10.2 2024-08-20T22:21:15.8672181Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:15.8675110Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:15.8676703Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:15.8677211Z + install_tlparse 2024-08-20T22:21:15.8677769Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:15.8678336Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:16.2922329Z Collecting tlparse==0.3.25 2024-08-20T22:21:16.3100781Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:16.3486103Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:16.9878250Z Installing collected packages: tlparse 2024-08-20T22:21:17.0666462Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:17.1433866Z ++ python -m site --user-base 2024-08-20T22:21:17.1677898Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:17.1679612Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:17.1680369Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:17.1681082Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:17.1682161Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:17.1683396Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:17.1684166Z + cd test 2024-08-20T22:21:17.1684781Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:18.8049484Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:18.8050171Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:18.8052971Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:18.8053441Z + [[ dynamic_inductor_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:21:18.8054120Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-08-20T22:21:18.8054649Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-08-20T22:21:18.8055169Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T22:21:18.8055680Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-08-20T22:21:18.8056173Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-08-20T22:21:18.8056852Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:18.8057315Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-08-20T22:21:18.8057972Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:21:18.8058549Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:18.8059052Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:18.8087038Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:18.8087968Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:18.8090724Z + cd test 2024-08-20T22:21:18.8091385Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:20.2732198Z PyTorch built with: 2024-08-20T22:21:20.2732943Z - GCC 9.4 2024-08-20T22:21:20.2733332Z - C++ Version: 201703 2024-08-20T22:21:20.2734199Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:20.2735272Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:20.2735942Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:20.2736469Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:20.2736956Z - NNPACK is enabled 2024-08-20T22:21:20.2737336Z - CPU capability usage: AVX2 2024-08-20T22:21:20.2737752Z - CUDA Runtime 12.1 2024-08-20T22:21:20.2738278Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:20.2739274Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:20.2739710Z - Magma 2.6.1 2024-08-20T22:21:20.2746933Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:20.2753240Z 2024-08-20T22:21:20.5943218Z + cd test 2024-08-20T22:21:20.5944199Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:21.9459238Z ATen/Parallel: 2024-08-20T22:21:21.9459699Z at::get_num_threads() : 8 2024-08-20T22:21:21.9460180Z at::get_num_interop_threads() : 16 2024-08-20T22:21:21.9460612Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:21.9461016Z omp_get_max_threads() : 8 2024-08-20T22:21:21.9462307Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:21.9463114Z mkl_get_max_threads() : 8 2024-08-20T22:21:21.9463769Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:21.9464500Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:21.9464931Z Environment variables: 2024-08-20T22:21:21.9465274Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:21.9465657Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:21.9466039Z ATen parallel backend: OpenMP 2024-08-20T22:21:21.9466292Z 2024-08-20T22:21:22.2217154Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:22.2217806Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-08-20T22:21:22.2218321Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-08-20T22:21:22.2218825Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-08-20T22:21:22.2219386Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:22.2220050Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:22.2220612Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-08-20T22:21:22.2221205Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-08-20T22:21:22.2221848Z + [[ dynamic_inductor_torchbench == *inductor-halide* ]] 2024-08-20T22:21:22.2222516Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:21:22.2223121Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-08-20T22:21:22.2223624Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-08-20T22:21:22.2224247Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-08-20T22:21:22.2224750Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:22.2225190Z + install_torchaudio cuda 2024-08-20T22:21:22.2225529Z + local commit 2024-08-20T22:21:22.2225835Z ++ get_pinned_commit audio 2024-08-20T22:21:22.2226201Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:21:22.2240792Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.2241263Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:21:22.2241682Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:21:22.2243039Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.2244485Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.5596189Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.5600027Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-jaueu0yp 2024-08-20T22:21:22.5631282Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-jaueu0yp 2024-08-20T22:21:23.3218963Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:21:23.3244014Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:23.7263603Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:23.7264840Z Running command git submodule update --init --recursive -q 2024-08-20T22:21:26.1972375Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:21:26.2000077Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:26.2027519Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:21:26.2031799Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:21:26.2033958Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:21:26.2036519Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:21:26.2039986Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:21:26.2044591Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:21:26.2058630Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:21:26.2534189Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:21:26.2596433Z Building wheels for collected packages: torchaudio 2024-08-20T22:23:57.7894387Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:23:57.7929829Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310391 sha256=4f949ab424ac8736aa2048959cf6036251bda43e351736882aa36478dba1fded 2024-08-20T22:23:57.7933677Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:23:57.7961551Z Successfully built torchaudio 2024-08-20T22:23:58.2885879Z Installing collected packages: torchaudio 2024-08-20T22:23:58.5045843Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:23:58.6855806Z + install_torchtext 2024-08-20T22:23:58.6856203Z + local data_commit 2024-08-20T22:23:58.6856522Z + local text_commit 2024-08-20T22:23:58.6859450Z ++ get_pinned_commit data 2024-08-20T22:23:58.6862264Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:23:58.6883688Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:58.6887494Z ++ get_pinned_commit text 2024-08-20T22:23:58.6887945Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:23:58.6906636Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:23:58.6908133Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:58.6909587Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:59.0286299Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:59.0290793Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-44mo0har 2024-08-20T22:23:59.0335445Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-44mo0har 2024-08-20T22:23:59.4948915Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:23:59.4977845Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:59.7508783Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:59.9013223Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:59.9014351Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:20.4184918Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:20.4222443Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:25:20.4225035Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:25:20.4230010Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:20.4260897Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:25:20.4264963Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:25:20.4268831Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:25:20.4271574Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:25:20.4276151Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:25:20.4281259Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:25:20.4296228Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:25:20.4355820Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:25:20.4359646Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:25:20.4364947Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:25:20.4869241Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:25:20.4934541Z Building wheels for collected packages: torchdata 2024-08-20T22:25:22.4436913Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:25:22.4444853Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=9b0979cc3c40036231bd8afbc02f35ad08ff3ce8ed21e9bb3665cdd9c1b5a75a 2024-08-20T22:25:22.4448068Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:25:22.4474324Z Successfully built torchdata 2024-08-20T22:25:22.9339833Z Installing collected packages: torchdata 2024-08-20T22:25:23.0294825Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:25:28.0130146Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:28.0131645Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:28.3483806Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:28.3488648Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-nr6eakyz 2024-08-20T22:25:28.3518742Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-nr6eakyz 2024-08-20T22:25:29.3431955Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:25:29.3461913Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:29.8430408Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:30.0096863Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:30.0097975Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:37.4864296Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:37.4899538Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:25:37.4902631Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:25:37.4906480Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:37.4909670Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:25:37.4913244Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:25:37.4939916Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:25:37.4943853Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:25:37.4948337Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:25:37.4952635Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:25:37.4980590Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:25:37.4984868Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:25:37.4988272Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:25:37.4992056Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:25:37.4995176Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:25:37.5000420Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:25:37.5015271Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:25:37.5602705Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:25:37.5669203Z Building wheels for collected packages: torchtext 2024-08-20T22:26:15.8306131Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:26:15.8338837Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045465 sha256=fbbd4de0a30493233df281ecadd914aea32b908fbea85e086e06965688fde082 2024-08-20T22:26:15.8342459Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:26:15.8368642Z Successfully built torchtext 2024-08-20T22:26:16.3245834Z Installing collected packages: torchtext 2024-08-20T22:26:16.4391176Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:26:16.5997373Z + install_torchvision 2024-08-20T22:26:16.5997748Z + local orig_preload 2024-08-20T22:26:16.5998072Z + local commit 2024-08-20T22:26:16.6001805Z ++ get_pinned_commit vision 2024-08-20T22:26:16.6002201Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:26:16.6020120Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:16.6020576Z + orig_preload= 2024-08-20T22:26:16.6020979Z + '[' -n '' ']' 2024-08-20T22:26:16.6024726Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:16.6026234Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:16.9397864Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:16.9403211Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-onimli14 2024-08-20T22:26:16.9434297Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-onimli14 2024-08-20T22:26:18.4356283Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:26:18.4385420Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:19.7722303Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:20.0890035Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:22.6024090Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:26:22.6061623Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:26:22.6065657Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:26:22.6070296Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:26:22.6143435Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:26:22.6149164Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:26:22.6152655Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:26:22.6154627Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:26:22.6156254Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:26:22.6160633Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:26:22.6175108Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:26:22.6653879Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:26:22.6717590Z Building wheels for collected packages: torchvision 2024-08-20T22:27:39.5697676Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:27:39.5733177Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116150 sha256=7e38fc33c609bfd2f5691f0009fe2bce4f4b01f4c00ca6f2ea71a81312aad75d 2024-08-20T22:27:39.5735873Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:27:39.5770674Z Successfully built torchvision 2024-08-20T22:27:40.0697270Z Installing collected packages: torchvision 2024-08-20T22:27:40.4885615Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:27:40.6324263Z + '[' -n '' ']' 2024-08-20T22:27:40.6325295Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:27:40.6328552Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:27:40.6329454Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:27:40.9693326Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:27:40.9697870Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-d_qwvf8s 2024-08-20T22:27:40.9728340Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-d_qwvf8s 2024-08-20T22:27:41.5261484Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:27:43.9392428Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:27:43.9427325Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:29:02.3714819Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / done 2024-08-20T22:29:02.3732574Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928017 sha256=4f879c5bfe8557dcc8ba10c88c3bd785b1cc8f02beb999d0727491556bc48250 2024-08-20T22:29:02.3735798Z Stored in directory: /tmp/pip-ephem-wheel-cache-_zzokk0l/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:29:02.3764586Z Successfully built torchao 2024-08-20T22:29:02.9429386Z Installing collected packages: torchao 2024-08-20T22:29:03.1834248Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:29:03.2931901Z + id=1 2024-08-20T22:29:03.2932538Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:29:03.2935242Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:29:03.7581944Z Collecting opencv-python==4.8.0.74 2024-08-20T22:29:03.7767132Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:29:03.7857549Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:29:03.7923910Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:29:04.8591734Z Installing collected packages: opencv-python 2024-08-20T22:29:05.7243630Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:29:05.8224223Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:29:05.8225006Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:29:05.8225852Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:29:05.8226387Z + checkout_install_torchbench 2024-08-20T22:29:05.8226740Z + local commit 2024-08-20T22:29:05.8231822Z ++ get_pinned_commit torchbench 2024-08-20T22:29:05.8232379Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:29:05.8249383Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:05.8250008Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:29:05.8266417Z Cloning into 'torchbench'... 2024-08-20T22:29:05.9608833Z remote: Enumerating objects: 31006, done. 2024-08-20T22:29:05.9610831Z remote: Counting objects: 0% (1/1746) 2024-08-20T22:29:05.9613786Z remote: Counting objects: 1% (18/1746) 2024-08-20T22:29:05.9615187Z remote: Counting objects: 2% (35/1746) 2024-08-20T22:29:05.9616007Z remote: Counting objects: 3% (53/1746) 2024-08-20T22:29:05.9616778Z remote: Counting objects: 4% (70/1746) 2024-08-20T22:29:05.9617564Z remote: Counting objects: 5% (88/1746) 2024-08-20T22:29:05.9618323Z remote: Counting objects: 6% (105/1746) 2024-08-20T22:29:05.9619109Z remote: Counting objects: 7% (123/1746) 2024-08-20T22:29:05.9622210Z remote: Counting objects: 8% (140/1746) 2024-08-20T22:29:05.9623019Z remote: Counting objects: 9% (158/1746) 2024-08-20T22:29:05.9623736Z remote: Counting objects: 10% (175/1746) 2024-08-20T22:29:05.9624597Z remote: Counting objects: 11% (193/1746) 2024-08-20T22:29:05.9625285Z remote: Counting objects: 12% (210/1746) 2024-08-20T22:29:05.9626025Z remote: Counting objects: 13% (227/1746) 2024-08-20T22:29:05.9626648Z remote: Counting objects: 14% (245/1746) 2024-08-20T22:29:05.9627367Z remote: Counting objects: 15% (262/1746) 2024-08-20T22:29:05.9628055Z remote: Counting objects: 16% (280/1746) 2024-08-20T22:29:05.9628696Z remote: Counting objects: 17% (297/1746) 2024-08-20T22:29:05.9629381Z remote: Counting objects: 18% (315/1746) 2024-08-20T22:29:05.9629930Z remote: Counting objects: 19% (332/1746) 2024-08-20T22:29:05.9630468Z remote: Counting objects: 20% (350/1746) 2024-08-20T22:29:05.9630994Z remote: Counting objects: 21% (367/1746) 2024-08-20T22:29:05.9631515Z remote: Counting objects: 22% (385/1746) 2024-08-20T22:29:05.9632039Z remote: Counting objects: 23% (402/1746) 2024-08-20T22:29:05.9632824Z remote: Counting objects: 24% (420/1746) 2024-08-20T22:29:05.9633347Z remote: Counting objects: 25% (437/1746) 2024-08-20T22:29:05.9633868Z remote: Counting objects: 26% (454/1746) 2024-08-20T22:29:05.9634380Z remote: Counting objects: 27% (472/1746) 2024-08-20T22:29:05.9635880Z remote: Counting objects: 28% (489/1746) 2024-08-20T22:29:05.9636417Z remote: Counting objects: 29% (507/1746) 2024-08-20T22:29:05.9636945Z remote: Counting objects: 30% (524/1746) 2024-08-20T22:29:05.9637457Z remote: Counting objects: 31% (542/1746) 2024-08-20T22:29:05.9637972Z remote: Counting objects: 32% (559/1746) 2024-08-20T22:29:05.9638487Z remote: Counting objects: 33% (577/1746) 2024-08-20T22:29:05.9639006Z remote: Counting objects: 34% (594/1746) 2024-08-20T22:29:05.9639519Z remote: Counting objects: 35% (612/1746) 2024-08-20T22:29:05.9640032Z remote: Counting objects: 36% (629/1746) 2024-08-20T22:29:05.9640548Z remote: Counting objects: 37% (647/1746) 2024-08-20T22:29:05.9641065Z remote: Counting objects: 38% (664/1746) 2024-08-20T22:29:05.9641607Z remote: Counting objects: 39% (681/1746) 2024-08-20T22:29:05.9642116Z remote: Counting objects: 40% (699/1746) 2024-08-20T22:29:05.9642624Z remote: Counting objects: 41% (716/1746) 2024-08-20T22:29:05.9643134Z remote: Counting objects: 42% (734/1746) 2024-08-20T22:29:05.9643647Z remote: Counting objects: 43% (751/1746) 2024-08-20T22:29:05.9644158Z remote: Counting objects: 44% (769/1746) 2024-08-20T22:29:05.9644670Z remote: Counting objects: 45% (786/1746) 2024-08-20T22:29:05.9645170Z remote: Counting objects: 46% (804/1746) 2024-08-20T22:29:05.9645683Z remote: Counting objects: 47% (821/1746) 2024-08-20T22:29:05.9646193Z remote: Counting objects: 48% (839/1746) 2024-08-20T22:29:05.9646703Z remote: Counting objects: 49% (856/1746) 2024-08-20T22:29:05.9647211Z remote: Counting objects: 50% (873/1746) 2024-08-20T22:29:05.9647722Z remote: Counting objects: 51% (891/1746) 2024-08-20T22:29:05.9648239Z remote: Counting objects: 52% (908/1746) 2024-08-20T22:29:05.9648751Z remote: Counting objects: 53% (926/1746) 2024-08-20T22:29:05.9649263Z remote: Counting objects: 54% (943/1746) 2024-08-20T22:29:05.9649774Z remote: Counting objects: 55% (961/1746) 2024-08-20T22:29:05.9650292Z remote: Counting objects: 56% (978/1746) 2024-08-20T22:29:05.9650810Z remote: Counting objects: 57% (996/1746) 2024-08-20T22:29:05.9651379Z remote: Counting objects: 58% (1013/1746) 2024-08-20T22:29:05.9651908Z remote: Counting objects: 59% (1031/1746) 2024-08-20T22:29:05.9652433Z remote: Counting objects: 60% (1048/1746) 2024-08-20T22:29:05.9652951Z remote: Counting objects: 61% (1066/1746) 2024-08-20T22:29:05.9653475Z remote: Counting objects: 62% (1083/1746) 2024-08-20T22:29:05.9654000Z remote: Counting objects: 63% (1100/1746) 2024-08-20T22:29:05.9654520Z remote: Counting objects: 64% (1118/1746) 2024-08-20T22:29:05.9655041Z remote: Counting objects: 65% (1135/1746) 2024-08-20T22:29:05.9655560Z remote: Counting objects: 66% (1153/1746) 2024-08-20T22:29:05.9656081Z remote: Counting objects: 67% (1170/1746) 2024-08-20T22:29:05.9656598Z remote: Counting objects: 68% (1188/1746) 2024-08-20T22:29:05.9657118Z remote: Counting objects: 69% (1205/1746) 2024-08-20T22:29:05.9657633Z remote: Counting objects: 70% (1223/1746) 2024-08-20T22:29:05.9658155Z remote: Counting objects: 71% (1240/1746) 2024-08-20T22:29:05.9658681Z remote: Counting objects: 72% (1258/1746) 2024-08-20T22:29:05.9659194Z remote: Counting objects: 73% (1275/1746) 2024-08-20T22:29:05.9659711Z remote: Counting objects: 74% (1293/1746) 2024-08-20T22:29:05.9660236Z remote: Counting objects: 75% (1310/1746) 2024-08-20T22:29:05.9660759Z remote: Counting objects: 76% (1327/1746) 2024-08-20T22:29:05.9661273Z remote: Counting objects: 77% (1345/1746) 2024-08-20T22:29:05.9661794Z remote: Counting objects: 78% (1362/1746) 2024-08-20T22:29:05.9662419Z remote: Counting objects: 79% (1380/1746) 2024-08-20T22:29:05.9662928Z remote: Counting objects: 80% (1397/1746) 2024-08-20T22:29:05.9663452Z remote: Counting objects: 81% (1415/1746) 2024-08-20T22:29:05.9663969Z remote: Counting objects: 82% (1432/1746) 2024-08-20T22:29:05.9664698Z remote: Counting objects: 83% (1450/1746) 2024-08-20T22:29:05.9665213Z remote: Counting objects: 84% (1467/1746) 2024-08-20T22:29:05.9665731Z remote: Counting objects: 85% (1485/1746) 2024-08-20T22:29:05.9666246Z remote: Counting objects: 86% (1502/1746) 2024-08-20T22:29:05.9666763Z remote: Counting objects: 87% (1520/1746) 2024-08-20T22:29:05.9667272Z remote: Counting objects: 88% (1537/1746) 2024-08-20T22:29:05.9667789Z remote: Counting objects: 89% (1554/1746) 2024-08-20T22:29:05.9668307Z remote: Counting objects: 90% (1572/1746) 2024-08-20T22:29:05.9668823Z remote: Counting objects: 91% (1589/1746) 2024-08-20T22:29:05.9669334Z remote: Counting objects: 92% (1607/1746) 2024-08-20T22:29:05.9669868Z remote: Counting objects: 93% (1624/1746) 2024-08-20T22:29:05.9670382Z remote: Counting objects: 94% (1642/1746) 2024-08-20T22:29:05.9670900Z remote: Counting objects: 95% (1659/1746) 2024-08-20T22:29:05.9671411Z remote: Counting objects: 96% (1677/1746) 2024-08-20T22:29:05.9672286Z remote: Counting objects: 97% (1694/1746) 2024-08-20T22:29:05.9672830Z remote: Counting objects: 98% (1712/1746) 2024-08-20T22:29:05.9673349Z remote: Counting objects: 99% (1729/1746) 2024-08-20T22:29:05.9673876Z remote: Counting objects: 100% (1746/1746) 2024-08-20T22:29:05.9674435Z remote: Counting objects: 100% (1746/1746), done. 2024-08-20T22:29:05.9675013Z remote: Compressing objects: 0% (1/902) 2024-08-20T22:29:05.9730766Z remote: Compressing objects: 1% (10/902) 2024-08-20T22:29:05.9854094Z remote: Compressing objects: 2% (19/902) 2024-08-20T22:29:05.9941187Z remote: Compressing objects: 3% (28/902) 2024-08-20T22:29:05.9992222Z remote: Compressing objects: 4% (37/902) 2024-08-20T22:29:06.0045329Z remote: Compressing objects: 5% (46/902) 2024-08-20T22:29:06.0078500Z remote: Compressing objects: 6% (55/902) 2024-08-20T22:29:06.0106332Z remote: Compressing objects: 7% (64/902) 2024-08-20T22:29:06.0147309Z remote: Compressing objects: 8% (73/902) 2024-08-20T22:29:06.0180449Z remote: Compressing objects: 9% (82/902) 2024-08-20T22:29:06.0209594Z remote: Compressing objects: 10% (91/902) 2024-08-20T22:29:06.0222432Z remote: Compressing objects: 11% (100/902) 2024-08-20T22:29:06.0233842Z remote: Compressing objects: 12% (109/902) 2024-08-20T22:29:06.0250539Z remote: Compressing objects: 13% (118/902) 2024-08-20T22:29:06.0266776Z remote: Compressing objects: 14% (127/902) 2024-08-20T22:29:06.0275621Z remote: Compressing objects: 15% (136/902) 2024-08-20T22:29:06.0284877Z remote: Compressing objects: 16% (145/902) 2024-08-20T22:29:06.0291265Z remote: Compressing objects: 17% (154/902) 2024-08-20T22:29:06.0294413Z remote: Compressing objects: 18% (163/902) 2024-08-20T22:29:06.0297722Z remote: Compressing objects: 19% (172/902) 2024-08-20T22:29:06.0299645Z remote: Compressing objects: 20% (181/902) 2024-08-20T22:29:06.0300421Z remote: Compressing objects: 21% (190/902) 2024-08-20T22:29:06.0302316Z remote: Compressing objects: 22% (199/902) 2024-08-20T22:29:06.0304976Z remote: Compressing objects: 23% (208/902) 2024-08-20T22:29:06.0306911Z remote: Compressing objects: 24% (217/902) 2024-08-20T22:29:06.0307473Z remote: Compressing objects: 25% (226/902) 2024-08-20T22:29:06.0309826Z remote: Compressing objects: 26% (235/902) 2024-08-20T22:29:06.0317991Z remote: Compressing objects: 27% (244/902) 2024-08-20T22:29:06.0320740Z remote: Compressing objects: 28% (253/902) 2024-08-20T22:29:06.0324269Z remote: Compressing objects: 29% (262/902) 2024-08-20T22:29:06.0327348Z remote: Compressing objects: 30% (271/902) 2024-08-20T22:29:06.0330281Z remote: Compressing objects: 31% (280/902) 2024-08-20T22:29:06.0333598Z remote: Compressing objects: 32% (289/902) 2024-08-20T22:29:06.0337470Z remote: Compressing objects: 33% (298/902) 2024-08-20T22:29:06.0341198Z remote: Compressing objects: 34% (307/902) 2024-08-20T22:29:06.0343350Z remote: Compressing objects: 35% (316/902) 2024-08-20T22:29:06.0345966Z remote: Compressing objects: 36% (325/902) 2024-08-20T22:29:06.0348555Z remote: Compressing objects: 37% (334/902) 2024-08-20T22:29:06.0350884Z remote: Compressing objects: 38% (343/902) 2024-08-20T22:29:06.0352403Z remote: Compressing objects: 39% (352/902) 2024-08-20T22:29:06.0353820Z remote: Compressing objects: 40% (361/902) 2024-08-20T22:29:06.0356016Z remote: Compressing objects: 41% (370/902) 2024-08-20T22:29:06.0357105Z remote: Compressing objects: 42% (379/902) 2024-08-20T22:29:06.0358187Z remote: Compressing objects: 43% (388/902) 2024-08-20T22:29:06.0359593Z remote: Compressing objects: 44% (397/902) 2024-08-20T22:29:06.0360882Z remote: Compressing objects: 45% (406/902) 2024-08-20T22:29:06.0361511Z remote: Compressing objects: 46% (415/902) 2024-08-20T22:29:06.0362112Z remote: Compressing objects: 47% (424/902) 2024-08-20T22:29:06.0362798Z remote: Compressing objects: 48% (433/902) 2024-08-20T22:29:06.0363394Z remote: Compressing objects: 49% (442/902) 2024-08-20T22:29:06.0363945Z remote: Compressing objects: 50% (451/902) 2024-08-20T22:29:06.0364557Z remote: Compressing objects: 51% (461/902) 2024-08-20T22:29:06.0365231Z remote: Compressing objects: 52% (470/902) 2024-08-20T22:29:06.0365877Z remote: Compressing objects: 53% (479/902) 2024-08-20T22:29:06.0366425Z remote: Compressing objects: 54% (488/902) 2024-08-20T22:29:06.0367113Z remote: Compressing objects: 55% (497/902) 2024-08-20T22:29:06.0367664Z remote: Compressing objects: 56% (506/902) 2024-08-20T22:29:06.0368234Z remote: Compressing objects: 57% (515/902) 2024-08-20T22:29:06.0368891Z remote: Compressing objects: 58% (524/902) 2024-08-20T22:29:06.0369545Z remote: Compressing objects: 59% (533/902) 2024-08-20T22:29:06.0370087Z remote: Compressing objects: 60% (542/902) 2024-08-20T22:29:06.0370624Z remote: Compressing objects: 61% (551/902) 2024-08-20T22:29:06.0371165Z remote: Compressing objects: 62% (560/902) 2024-08-20T22:29:06.0371694Z remote: Compressing objects: 63% (569/902) 2024-08-20T22:29:06.0372499Z remote: Compressing objects: 64% (578/902) 2024-08-20T22:29:06.0373041Z remote: Compressing objects: 65% (587/902) 2024-08-20T22:29:06.0373638Z remote: Compressing objects: 66% (596/902) 2024-08-20T22:29:06.0374237Z remote: Compressing objects: 67% (605/902) 2024-08-20T22:29:06.0374769Z remote: Compressing objects: 68% (614/902) 2024-08-20T22:29:06.0375308Z remote: Compressing objects: 69% (623/902) 2024-08-20T22:29:06.0375843Z remote: Compressing objects: 70% (632/902) 2024-08-20T22:29:06.0376381Z remote: Compressing objects: 71% (641/902) 2024-08-20T22:29:06.0376927Z remote: Compressing objects: 72% (650/902) 2024-08-20T22:29:06.0377465Z remote: Compressing objects: 73% (659/902) 2024-08-20T22:29:06.0377985Z remote: Compressing objects: 74% (668/902) 2024-08-20T22:29:06.0378514Z remote: Compressing objects: 75% (677/902) 2024-08-20T22:29:06.0383524Z remote: Compressing objects: 76% (686/902) 2024-08-20T22:29:06.0391208Z remote: Compressing objects: 77% (695/902) 2024-08-20T22:29:06.0395735Z remote: Compressing objects: 78% (704/902) 2024-08-20T22:29:06.0401790Z remote: Compressing objects: 79% (713/902) 2024-08-20T22:29:06.0408348Z remote: Compressing objects: 80% (722/902) 2024-08-20T22:29:06.0417673Z remote: Compressing objects: 81% (731/902) 2024-08-20T22:29:06.0424514Z remote: Compressing objects: 82% (740/902) 2024-08-20T22:29:06.0428084Z remote: Compressing objects: 83% (749/902) 2024-08-20T22:29:06.0431925Z remote: Compressing objects: 84% (758/902) 2024-08-20T22:29:06.0435936Z remote: Compressing objects: 85% (767/902) 2024-08-20T22:29:06.0440812Z remote: Compressing objects: 86% (776/902) 2024-08-20T22:29:06.0445062Z remote: Compressing objects: 87% (785/902) 2024-08-20T22:29:06.0448802Z remote: Compressing objects: 88% (794/902) 2024-08-20T22:29:06.0453822Z remote: Compressing objects: 89% (803/902) 2024-08-20T22:29:06.0456156Z remote: Compressing objects: 90% (812/902) 2024-08-20T22:29:06.0459922Z remote: Compressing objects: 91% (821/902) 2024-08-20T22:29:06.0462892Z remote: Compressing objects: 92% (830/902) 2024-08-20T22:29:06.0466058Z remote: Compressing objects: 93% (839/902) 2024-08-20T22:29:06.0468677Z remote: Compressing objects: 94% (848/902) 2024-08-20T22:29:06.0471243Z remote: Compressing objects: 95% (857/902) 2024-08-20T22:29:06.0474751Z remote: Compressing objects: 96% (866/902) 2024-08-20T22:29:06.0477290Z remote: Compressing objects: 97% (875/902) 2024-08-20T22:29:06.0478222Z remote: Compressing objects: 98% (884/902) 2024-08-20T22:29:06.0480123Z remote: Compressing objects: 99% (893/902) 2024-08-20T22:29:06.0481278Z remote: Compressing objects: 100% (902/902) 2024-08-20T22:29:06.0481846Z remote: Compressing objects: 100% (902/902), done. 2024-08-20T22:29:06.0561037Z Receiving objects: 0% (1/31006) 2024-08-20T22:29:06.0603801Z Receiving objects: 1% (311/31006) 2024-08-20T22:29:06.0643322Z Receiving objects: 2% (621/31006) 2024-08-20T22:29:06.0682767Z Receiving objects: 3% (931/31006) 2024-08-20T22:29:06.0716354Z Receiving objects: 4% (1241/31006) 2024-08-20T22:29:06.0749084Z Receiving objects: 5% (1551/31006) 2024-08-20T22:29:06.0786788Z Receiving objects: 6% (1861/31006) 2024-08-20T22:29:06.0825662Z Receiving objects: 7% (2171/31006) 2024-08-20T22:29:06.0855920Z Receiving objects: 8% (2481/31006) 2024-08-20T22:29:06.0889198Z Receiving objects: 9% (2791/31006) 2024-08-20T22:29:06.0959136Z Receiving objects: 10% (3101/31006) 2024-08-20T22:29:06.1007846Z Receiving objects: 11% (3411/31006) 2024-08-20T22:29:06.1073791Z Receiving objects: 12% (3721/31006) 2024-08-20T22:29:06.1205397Z Receiving objects: 13% (4031/31006) 2024-08-20T22:29:06.1349156Z Receiving objects: 14% (4341/31006) 2024-08-20T22:29:06.1420894Z Receiving objects: 15% (4651/31006) 2024-08-20T22:29:06.1494505Z Receiving objects: 16% (4961/31006) 2024-08-20T22:29:06.1554742Z Receiving objects: 17% (5272/31006) 2024-08-20T22:29:06.1583077Z Receiving objects: 18% (5582/31006) 2024-08-20T22:29:06.1641098Z Receiving objects: 19% (5892/31006) 2024-08-20T22:29:06.1674980Z Receiving objects: 20% (6202/31006) 2024-08-20T22:29:06.1692151Z Receiving objects: 21% (6512/31006) 2024-08-20T22:29:06.1908707Z Receiving objects: 22% (6822/31006) 2024-08-20T22:29:07.0507215Z Receiving objects: 23% (7132/31006) 2024-08-20T22:29:07.2241556Z Receiving objects: 23% (7375/31006), 72.12 MiB | 72.12 MiB/s 2024-08-20T22:29:07.8191987Z Receiving objects: 24% (7442/31006), 72.12 MiB | 72.12 MiB/s 2024-08-20T22:29:07.8959957Z Receiving objects: 25% (7752/31006), 107.90 MiB | 71.93 MiB/s 2024-08-20T22:29:07.9731190Z Receiving objects: 26% (8062/31006), 107.90 MiB | 71.93 MiB/s 2024-08-20T22:29:08.0484265Z Receiving objects: 27% (8372/31006), 107.90 MiB | 71.93 MiB/s 2024-08-20T22:29:08.0505435Z Receiving objects: 28% (8682/31006), 107.90 MiB | 71.93 MiB/s 2024-08-20T22:29:08.1245129Z Receiving objects: 28% (8688/31006), 142.25 MiB | 71.12 MiB/s 2024-08-20T22:29:08.2002865Z Receiving objects: 29% (8992/31006), 142.25 MiB | 71.12 MiB/s 2024-08-20T22:29:08.2759253Z Receiving objects: 30% (9302/31006), 142.25 MiB | 71.12 MiB/s 2024-08-20T22:29:08.3521602Z Receiving objects: 31% (9612/31006), 142.25 MiB | 71.12 MiB/s 2024-08-20T22:29:08.5973595Z Receiving objects: 32% (9922/31006), 142.25 MiB | 71.12 MiB/s 2024-08-20T22:29:08.6736682Z Receiving objects: 33% (10232/31006), 178.57 MiB | 71.43 MiB/s 2024-08-20T22:29:08.6870112Z Receiving objects: 34% (10543/31006), 178.57 MiB | 71.43 MiB/s 2024-08-20T22:29:08.9580304Z Receiving objects: 35% (10853/31006), 178.57 MiB | 71.43 MiB/s 2024-08-20T22:29:09.0506960Z Receiving objects: 36% (11163/31006), 178.57 MiB | 71.43 MiB/s 2024-08-20T22:29:09.2246679Z Receiving objects: 36% (11222/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2265105Z Receiving objects: 37% (11473/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2289789Z Receiving objects: 38% (11783/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2310883Z Receiving objects: 39% (12093/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2326111Z Receiving objects: 40% (12403/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2337383Z Receiving objects: 41% (12713/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2350703Z Receiving objects: 42% (13023/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2366619Z Receiving objects: 43% (13333/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2383302Z Receiving objects: 44% (13643/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2392085Z Receiving objects: 45% (13953/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2397133Z Receiving objects: 46% (14263/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2410994Z Receiving objects: 47% (14573/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2433438Z Receiving objects: 48% (14883/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2445991Z Receiving objects: 49% (15193/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2651762Z Receiving objects: 50% (15503/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2659199Z Receiving objects: 51% (15814/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2666617Z Receiving objects: 52% (16124/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2674181Z Receiving objects: 53% (16434/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2680474Z Receiving objects: 54% (16744/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2700084Z Receiving objects: 55% (17054/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2711801Z Receiving objects: 56% (17364/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2717987Z Receiving objects: 57% (17674/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2731748Z Receiving objects: 58% (17984/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2743843Z Receiving objects: 59% (18294/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2754522Z Receiving objects: 60% (18604/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2901247Z Receiving objects: 61% (18914/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2907597Z Receiving objects: 62% (19224/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2913769Z Receiving objects: 63% (19534/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2921960Z Receiving objects: 64% (19844/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2930009Z Receiving objects: 65% (20154/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.2938542Z Receiving objects: 66% (20464/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3207934Z Receiving objects: 67% (20775/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3615944Z Receiving objects: 68% (21085/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3624357Z Receiving objects: 69% (21395/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3637969Z Receiving objects: 70% (21705/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3649239Z Receiving objects: 71% (22015/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3664260Z Receiving objects: 72% (22325/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3673788Z Receiving objects: 73% (22635/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3691177Z Receiving objects: 74% (22945/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3700988Z Receiving objects: 75% (23255/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3708837Z Receiving objects: 76% (23565/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3717757Z Receiving objects: 77% (23875/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3935104Z Receiving objects: 78% (24185/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3969567Z Receiving objects: 79% (24495/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3979057Z Receiving objects: 80% (24805/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.3991598Z Receiving objects: 81% (25115/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4004483Z Receiving objects: 82% (25425/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4053611Z Receiving objects: 83% (25735/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4202271Z Receiving objects: 84% (26046/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4209183Z Receiving objects: 85% (26356/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4216582Z Receiving objects: 86% (26666/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4222054Z Receiving objects: 87% (26976/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4230641Z Receiving objects: 88% (27286/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4236689Z Receiving objects: 89% (27596/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4350982Z Receiving objects: 90% (27906/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4363917Z Receiving objects: 91% (28216/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4376534Z Receiving objects: 92% (28526/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4393338Z Receiving objects: 93% (28836/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4427210Z Receiving objects: 94% (29146/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4437551Z Receiving objects: 95% (29456/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4444377Z Receiving objects: 96% (29766/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4450579Z Receiving objects: 97% (30076/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4517971Z Receiving objects: 98% (30386/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4545211Z Receiving objects: 99% (30696/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4546290Z remote: Total 31006 (delta 1014), reused 1480 (delta 825), pack-reused 29260 (from 1) 2024-08-20T22:29:09.4562515Z Receiving objects: 100% (31006/31006), 207.50 MiB | 69.16 MiB/s 2024-08-20T22:29:09.4563210Z Receiving objects: 100% (31006/31006), 232.50 MiB | 68.28 MiB/s, done. 2024-08-20T22:29:09.4594442Z Resolving deltas: 0% (0/16734) 2024-08-20T22:29:09.4607188Z Resolving deltas: 1% (168/16734) 2024-08-20T22:29:09.4618117Z Resolving deltas: 2% (391/16734) 2024-08-20T22:29:09.4644249Z Resolving deltas: 3% (503/16734) 2024-08-20T22:29:09.4657876Z Resolving deltas: 4% (670/16734) 2024-08-20T22:29:09.4667279Z Resolving deltas: 5% (887/16734) 2024-08-20T22:29:09.4681351Z Resolving deltas: 6% (1008/16734) 2024-08-20T22:29:09.4690838Z Resolving deltas: 7% (1183/16734) 2024-08-20T22:29:09.4699936Z Resolving deltas: 8% (1361/16734) 2024-08-20T22:29:09.4710972Z Resolving deltas: 9% (1511/16734) 2024-08-20T22:29:09.4721630Z Resolving deltas: 10% (1684/16734) 2024-08-20T22:29:09.4734379Z Resolving deltas: 11% (1847/16734) 2024-08-20T22:29:09.4748933Z Resolving deltas: 12% (2010/16734) 2024-08-20T22:29:09.4760434Z Resolving deltas: 13% (2207/16734) 2024-08-20T22:29:09.4775788Z Resolving deltas: 14% (2384/16734) 2024-08-20T22:29:09.4793463Z Resolving deltas: 15% (2511/16734) 2024-08-20T22:29:09.4806926Z Resolving deltas: 16% (2690/16734) 2024-08-20T22:29:09.4844517Z Resolving deltas: 17% (2859/16734) 2024-08-20T22:29:09.4869620Z Resolving deltas: 18% (3032/16734) 2024-08-20T22:29:09.4890799Z Resolving deltas: 19% (3189/16734) 2024-08-20T22:29:09.4893355Z Resolving deltas: 20% (3486/16734) 2024-08-20T22:29:09.4905689Z Resolving deltas: 21% (3517/16734) 2024-08-20T22:29:09.4923358Z Resolving deltas: 22% (3685/16734) 2024-08-20T22:29:09.4935336Z Resolving deltas: 23% (3890/16734) 2024-08-20T22:29:09.4950567Z Resolving deltas: 24% (4046/16734) 2024-08-20T22:29:09.4988978Z Resolving deltas: 25% (4184/16734) 2024-08-20T22:29:09.5009847Z Resolving deltas: 26% (4370/16734) 2024-08-20T22:29:09.5022967Z Resolving deltas: 27% (4545/16734) 2024-08-20T22:29:09.5040060Z Resolving deltas: 28% (4690/16734) 2024-08-20T22:29:09.5057115Z Resolving deltas: 29% (4857/16734) 2024-08-20T22:29:09.5070845Z Resolving deltas: 30% (5021/16734) 2024-08-20T22:29:09.5083798Z Resolving deltas: 31% (5188/16734) 2024-08-20T22:29:09.5092612Z Resolving deltas: 32% (5388/16734) 2024-08-20T22:29:09.5108589Z Resolving deltas: 33% (5526/16734) 2024-08-20T22:29:09.5130715Z Resolving deltas: 34% (5704/16734) 2024-08-20T22:29:09.5150015Z Resolving deltas: 35% (5865/16734) 2024-08-20T22:29:09.5160798Z Resolving deltas: 36% (6049/16734) 2024-08-20T22:29:09.5182736Z Resolving deltas: 37% (6218/16734) 2024-08-20T22:29:09.5201160Z Resolving deltas: 38% (6359/16734) 2024-08-20T22:29:09.5213233Z Resolving deltas: 39% (6551/16734) 2024-08-20T22:29:09.5239165Z Resolving deltas: 40% (6696/16734) 2024-08-20T22:29:09.5245170Z Resolving deltas: 41% (6930/16734) 2024-08-20T22:29:09.5255786Z Resolving deltas: 42% (7043/16734) 2024-08-20T22:29:09.5265442Z Resolving deltas: 43% (7243/16734) 2024-08-20T22:29:09.5288600Z Resolving deltas: 44% (7373/16734) 2024-08-20T22:29:09.5301270Z Resolving deltas: 45% (7533/16734) 2024-08-20T22:29:09.5312228Z Resolving deltas: 46% (7703/16734) 2024-08-20T22:29:09.5328340Z Resolving deltas: 47% (7865/16734) 2024-08-20T22:29:09.5339441Z Resolving deltas: 48% (8034/16734) 2024-08-20T22:29:09.5345234Z Resolving deltas: 49% (8236/16734) 2024-08-20T22:29:09.5356212Z Resolving deltas: 50% (8371/16734) 2024-08-20T22:29:09.5376858Z Resolving deltas: 51% (8539/16734) 2024-08-20T22:29:09.5379111Z Resolving deltas: 52% (8836/16734) 2024-08-20T22:29:09.5386286Z Resolving deltas: 53% (8883/16734) 2024-08-20T22:29:09.5398610Z Resolving deltas: 54% (9046/16734) 2024-08-20T22:29:09.5409170Z Resolving deltas: 55% (9213/16734) 2024-08-20T22:29:09.5415973Z Resolving deltas: 56% (9375/16734) 2024-08-20T22:29:09.5426247Z Resolving deltas: 57% (9544/16734) 2024-08-20T22:29:09.5430748Z Resolving deltas: 58% (9795/16734) 2024-08-20T22:29:09.5445065Z Resolving deltas: 59% (9891/16734) 2024-08-20T22:29:09.5453562Z Resolving deltas: 61% (10245/16734) 2024-08-20T22:29:09.5460207Z Resolving deltas: 62% (10437/16734) 2024-08-20T22:29:09.5469435Z Resolving deltas: 63% (10614/16734) 2024-08-20T22:29:09.5488961Z Resolving deltas: 64% (10796/16734) 2024-08-20T22:29:09.5494872Z Resolving deltas: 66% (11081/16734) 2024-08-20T22:29:09.5516327Z Resolving deltas: 67% (11217/16734) 2024-08-20T22:29:09.5522832Z Resolving deltas: 68% (11381/16734) 2024-08-20T22:29:09.5534517Z Resolving deltas: 69% (11550/16734) 2024-08-20T22:29:09.5551231Z Resolving deltas: 70% (11721/16734) 2024-08-20T22:29:09.5564063Z Resolving deltas: 71% (11883/16734) 2024-08-20T22:29:09.5575104Z Resolving deltas: 72% (12069/16734) 2024-08-20T22:29:09.5602104Z Resolving deltas: 73% (12234/16734) 2024-08-20T22:29:09.5607214Z Resolving deltas: 75% (12628/16734) 2024-08-20T22:29:09.5620869Z Resolving deltas: 76% (12720/16734) 2024-08-20T22:29:09.5629607Z Resolving deltas: 77% (12896/16734) 2024-08-20T22:29:09.5646802Z Resolving deltas: 78% (13054/16734) 2024-08-20T22:29:09.5656526Z Resolving deltas: 79% (13228/16734) 2024-08-20T22:29:09.5669459Z Resolving deltas: 80% (13390/16734) 2024-08-20T22:29:09.5679292Z Resolving deltas: 81% (13580/16734) 2024-08-20T22:29:09.5692240Z Resolving deltas: 82% (13722/16734) 2024-08-20T22:29:09.5703504Z Resolving deltas: 83% (13897/16734) 2024-08-20T22:29:09.5711934Z Resolving deltas: 84% (14104/16734) 2024-08-20T22:29:09.5720120Z Resolving deltas: 85% (14246/16734) 2024-08-20T22:29:09.5730359Z Resolving deltas: 86% (14406/16734) 2024-08-20T22:29:09.5739376Z Resolving deltas: 87% (14559/16734) 2024-08-20T22:29:09.5750664Z Resolving deltas: 88% (14728/16734) 2024-08-20T22:29:09.5760195Z Resolving deltas: 89% (14906/16734) 2024-08-20T22:29:09.5769978Z Resolving deltas: 90% (15073/16734) 2024-08-20T22:29:09.5779434Z Resolving deltas: 91% (15237/16734) 2024-08-20T22:29:09.5784380Z Resolving deltas: 92% (15410/16734) 2024-08-20T22:29:09.5792139Z Resolving deltas: 93% (15564/16734) 2024-08-20T22:29:09.5804027Z Resolving deltas: 94% (15733/16734) 2024-08-20T22:29:09.5813609Z Resolving deltas: 95% (15959/16734) 2024-08-20T22:29:09.5822322Z Resolving deltas: 96% (16119/16734) 2024-08-20T22:29:09.5829716Z Resolving deltas: 97% (16249/16734) 2024-08-20T22:29:09.5885913Z Resolving deltas: 98% (16400/16734) 2024-08-20T22:29:09.5946299Z Resolving deltas: 99% (16588/16734) 2024-08-20T22:29:09.5946710Z Resolving deltas: 100% (16734/16734) 2024-08-20T22:29:09.5947134Z Resolving deltas: 100% (16734/16734), done. 2024-08-20T22:29:10.5973089Z + pushd torchbench 2024-08-20T22:29:10.5973500Z ~/workspace/torchbench ~/workspace 2024-08-20T22:29:10.5974011Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:10.6762992Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:29:10.6763547Z 2024-08-20T22:29:10.6764096Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:29:10.6764918Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:29:10.6765684Z state without impacting any branches by switching back to a branch. 2024-08-20T22:29:10.6766134Z 2024-08-20T22:29:10.6766445Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:29:10.6767264Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:29:10.6767685Z 2024-08-20T22:29:10.6767887Z git switch -c 2024-08-20T22:29:10.6768160Z 2024-08-20T22:29:10.6768299Z Or undo this operation with: 2024-08-20T22:29:10.6768552Z 2024-08-20T22:29:10.6768671Z git switch - 2024-08-20T22:29:10.6768850Z 2024-08-20T22:29:10.6769190Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:29:10.6769669Z 2024-08-20T22:29:10.6769840Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:29:10.6773141Z + '[' '' ']' 2024-08-20T22:29:10.6773556Z + python install.py --continue_on_fail 2024-08-20T22:29:12.9392964Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:29:31.1162229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:29:34.2078248Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:29:36.1163383Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:29:37.7551863Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:29:37.7782635Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:29:45.3843894Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:29:48.1156405Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:29:50.8432087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:29:53.5644901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:29:53.5647291Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:29:54.6204208Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:29:57.3952528Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:29:57.4186776Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:30:54.4577570Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:31:03.0525662Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:31:10.1843530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:31:16.9852229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:31:25.3861268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:31:32.3643717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:31:38.6855274Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:31:45.5547747Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:31:52.5441655Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:31:59.6513799Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:32:06.4820326Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:32:13.3389317Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:32:14.5530227Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:32:50.1593013Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:32:54.6909759Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:32:59.9170191Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:33:10.8157590Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:33:12.0068466Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:33:13.4903442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:33:19.9901393Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:33:27.0533605Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:33:32.5325772Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:33:40.7135516Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:33:46.6057851Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:33:51.6081408Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:33:57.7761791Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:34:14.2114901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:34:20.3899612Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:34:24.2860319Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:34:29.0177419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:34:35.7669308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:34:35.7672261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:34:48.7438312Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:34:52.6513250Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:34:52.6515823Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:35:00.2246459Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:35:01.4566763Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:35:02.7062684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:35:47.5143411Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:37:16.8119892Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:37:16.8398508Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:37:18.4043780Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:37:18.4280891Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:37:18.4521862Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:37:18.4766084Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:37:18.5006592Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:37:18.5250122Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:37:39.6094173Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:37:39.6097420Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:37:40.8709595Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:37:43.3001298Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:37:43.3241928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:37:43.3482883Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:37:43.3732665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:37:43.3976614Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:37:43.4215550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:37:50.0220287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:37:51.4643524Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:37:54.9880290Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:37:55.0124054Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:37:55.0371594Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:37:55.0609126Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:37:55.0849503Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:37:55.1086366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:38:10.2039728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:38:26.1929496Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:38:26.2172601Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:38:26.2178474Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:38:26.2182558Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:38:28.5627906Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:38:30.3268579Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:38:30.3511999Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:38:54.7139311Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:39:01.2840959Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:39:05.8296928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:39:12.9694745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:39:12.9940685Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:39:13.0187636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:39:13.0432870Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:39:13.0674222Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:39:13.0917524Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:39:13.1156946Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:39:13.1397255Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:39:22.4233879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:39:33.9394041Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:39:33.9644756Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:39:36.0703261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:39:38.0645128Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:39:38.5462293Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:39:38.5464012Z Print all dependencies after TorchBench is installed 2024-08-20T22:39:38.5465028Z + python -mpip freeze 2024-08-20T22:39:39.0285252Z absl-py==2.1.0 2024-08-20T22:39:39.0285620Z accelerate==0.33.0 2024-08-20T22:39:39.0286103Z aiohappyeyeballs==2.4.0 2024-08-20T22:39:39.0286516Z aiohttp==3.10.5 2024-08-20T22:39:39.0286932Z aiosignal==1.3.1 2024-08-20T22:39:39.0287372Z annotated-types==0.7.0 2024-08-20T22:39:39.0287932Z antlr4-python3-runtime==4.9.3 2024-08-20T22:39:39.0288440Z anyascii==0.3.2 2024-08-20T22:39:39.0293142Z astroid==3.2.4 2024-08-20T22:39:39.0293645Z astunparse==1.6.3 2024-08-20T22:39:39.0294326Z async-timeout==4.0.3 2024-08-20T22:39:39.0294784Z attrs==23.1.0 2024-08-20T22:39:39.0295175Z audioread==3.0.1 2024-08-20T22:39:39.0295684Z babel==2.16.0 2024-08-20T22:39:39.0296096Z beautifulsoup4==4.12.3 2024-08-20T22:39:39.0297529Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:39:39.0300321Z bibtexparser==2.0.0b7 2024-08-20T22:39:39.0300767Z black==24.8.0 2024-08-20T22:39:39.0301153Z blinker==1.8.2 2024-08-20T22:39:39.0301570Z blis==0.7.11 2024-08-20T22:39:39.0301959Z blobfile==2.1.1 2024-08-20T22:39:39.0302354Z bokeh==3.5.1 2024-08-20T22:39:39.0302728Z boto3==1.19.12 2024-08-20T22:39:39.0303384Z botocore==1.22.12 2024-08-20T22:39:39.0303808Z bs4==0.0.2 2024-08-20T22:39:39.0304159Z cachetools==5.5.0 2024-08-20T22:39:39.0304698Z cardboardlint==1.3.1 2024-08-20T22:39:39.0305111Z catalogue==2.0.10 2024-08-20T22:39:39.0305614Z certifi==2024.7.4 2024-08-20T22:39:39.0306011Z cffi==1.17.0 2024-08-20T22:39:39.0306501Z charset-normalizer==3.3.2 2024-08-20T22:39:39.0306943Z click==8.1.7 2024-08-20T22:39:39.0307319Z clldutils==3.22.2 2024-08-20T22:39:39.0307721Z cloudpathlib==0.18.1 2024-08-20T22:39:39.0308113Z cloudpickle==3.0.0 2024-08-20T22:39:39.0308424Z colorama==0.4.6 2024-08-20T22:39:39.0308720Z colorlog==6.8.2 2024-08-20T22:39:39.0309015Z confection==0.1.5 2024-08-20T22:39:39.0309322Z contourpy==1.2.1 2024-08-20T22:39:39.0309628Z coremltools==5.0b5 2024-08-20T22:39:39.0309941Z cryptography==43.0.0 2024-08-20T22:39:39.0310254Z csvw==3.3.0 2024-08-20T22:39:39.0310526Z cycler==0.12.1 2024-08-20T22:39:39.0310814Z cymem==2.0.8 2024-08-20T22:39:39.0311145Z DALL-E==0.1 2024-08-20T22:39:39.0311418Z datasets==2.21.0 2024-08-20T22:39:39.0311734Z decorator==5.1.1 2024-08-20T22:39:39.0312035Z defusedxml==0.7.1 2024-08-20T22:39:39.0312342Z Deprecated==1.2.14 2024-08-20T22:39:39.0313049Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:39:39.0313826Z diffusers==0.20.2 2024-08-20T22:39:39.0314126Z dill==0.3.8 2024-08-20T22:39:39.0314398Z diskcache==5.6.3 2024-08-20T22:39:39.0314698Z distro==1.9.0 2024-08-20T22:39:39.0314988Z dlinfo==1.2.1 2024-08-20T22:39:39.0315333Z docker-pycreds==0.4.0 2024-08-20T22:39:39.0315654Z dominate==2.9.1 2024-08-20T22:39:39.0316452Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:39:39.0317224Z einops==0.8.0 2024-08-20T22:39:39.0317527Z exceptiongroup==1.2.2 2024-08-20T22:39:39.0317846Z execnet==2.1.1 2024-08-20T22:39:39.0318171Z expecttest==0.1.6 2024-08-20T22:39:39.0318569Z FastNLP==0.6.0 2024-08-20T22:39:39.0318947Z ffmpeg-python==0.2.0 2024-08-20T22:39:39.0319279Z filelock==3.13.1 2024-08-20T22:39:39.0319580Z Flask==3.0.3 2024-08-20T22:39:39.0319860Z flatbuffers==2.0 2024-08-20T22:39:39.0320152Z fonttools==4.53.1 2024-08-20T22:39:39.0320453Z frozenlist==1.4.1 2024-08-20T22:39:39.0320751Z fsspec==2024.6.1 2024-08-20T22:39:39.0321031Z ftfy==6.2.3 2024-08-20T22:39:39.0321633Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:39:39.0322325Z future==1.0.0 2024-08-20T22:39:39.0322633Z fvcore==0.1.5.post20221221 2024-08-20T22:39:39.0322966Z gdown==5.2.0 2024-08-20T22:39:39.0323246Z geojson==2.5.0 2024-08-20T22:39:39.0323534Z ghstack==0.8.0 2024-08-20T22:39:39.0323813Z gitdb==4.0.11 2024-08-20T22:39:39.0324105Z GitPython==3.1.43 2024-08-20T22:39:39.0324452Z google-auth==2.34.0 2024-08-20T22:39:39.0324826Z google-auth-oauthlib==1.0.0 2024-08-20T22:39:39.0325179Z grpcio==1.65.5 2024-08-20T22:39:39.0325465Z gym==0.26.2 2024-08-20T22:39:39.0325782Z gym-notices==0.0.8 2024-08-20T22:39:39.0326088Z h5py==3.11.0 2024-08-20T22:39:39.0326365Z higher==0.2.1 2024-08-20T22:39:39.0326701Z huggingface-hub==0.24.6 2024-08-20T22:39:39.0327066Z hydra-core==1.3.2 2024-08-20T22:39:39.0327365Z hypothesis==5.35.1 2024-08-20T22:39:39.0327668Z idna==3.7 2024-08-20T22:39:39.0327940Z imageio==2.35.1 2024-08-20T22:39:39.0328246Z importlib_metadata==8.4.0 2024-08-20T22:39:39.0328576Z inflect==7.3.1 2024-08-20T22:39:39.0328874Z iniconfig==2.0.0 2024-08-20T22:39:39.0329169Z iopath==0.1.9 2024-08-20T22:39:39.0329448Z isodate==0.6.1 2024-08-20T22:39:39.0329737Z isort==5.13.2 2024-08-20T22:39:39.0330025Z itsdangerous==2.2.0 2024-08-20T22:39:39.0330332Z Jinja2==3.1.4 2024-08-20T22:39:39.0330610Z jmespath==0.10.0 2024-08-20T22:39:39.0331096Z joblib==1.4.2 2024-08-20T22:39:39.0331383Z jsonpatch==1.33 2024-08-20T22:39:39.0331674Z jsonpointer==3.0.0 2024-08-20T22:39:39.0331980Z jsonschema==4.23.0 2024-08-20T22:39:39.0332381Z jsonschema-specifications==2023.12.1 2024-08-20T22:39:39.0332789Z junitparser==2.1.1 2024-08-20T22:39:39.0333214Z kaldi-io==0.9.8 2024-08-20T22:39:39.0333515Z kiwisolver==1.4.5 2024-08-20T22:39:39.0333812Z kornia==0.7.3 2024-08-20T22:39:39.0334090Z kornia_rs==0.1.5 2024-08-20T22:39:39.0334382Z lameenc==1.7.0 2024-08-20T22:39:39.0334669Z langcodes==3.4.0 2024-08-20T22:39:39.0334967Z langdetect==1.0.9 2024-08-20T22:39:39.0335303Z language-tags==1.2.0 2024-08-20T22:39:39.0335625Z language_data==1.2.0 2024-08-20T22:39:39.0335932Z lark==0.12.0 2024-08-20T22:39:39.0336205Z lazy_loader==0.4 2024-08-20T22:39:39.0336496Z libcst==1.4.0 2024-08-20T22:39:39.0336781Z librosa==0.9.2 2024-08-20T22:39:39.0337070Z lintrunner==0.12.5 2024-08-20T22:39:39.0337362Z llvmlite==0.38.1 2024-08-20T22:39:39.0337649Z lxml==4.9.4 2024-08-20T22:39:39.0337967Z marisa-trie==1.2.0 2024-08-20T22:39:39.0338262Z Markdown==3.7 2024-08-20T22:39:39.0338589Z markdown-it-py==3.0.0 2024-08-20T22:39:39.0338911Z MarkupSafe==2.1.5 2024-08-20T22:39:39.0339215Z matplotlib==3.8.4 2024-08-20T22:39:39.0339506Z mccabe==0.7.0 2024-08-20T22:39:39.0339794Z mdurl==0.1.2 2024-08-20T22:39:39.0340115Z ml-dtypes==0.4.0 2024-08-20T22:39:39.0340411Z MonkeyType==23.3.0 2024-08-20T22:39:39.0340748Z more-itertools==10.4.0 2024-08-20T22:39:39.0341073Z mpmath==1.3.0 2024-08-20T22:39:39.0341396Z msgpack==1.0.8 2024-08-20T22:39:39.0341687Z multidict==6.0.5 2024-08-20T22:39:39.0341995Z multiprocess==0.70.16 2024-08-20T22:39:39.0342313Z murmurhash==1.0.10 2024-08-20T22:39:39.0342609Z musdb==0.4.2 2024-08-20T22:39:39.0342879Z museval==0.4.1 2024-08-20T22:39:39.0343159Z mypy==1.10.0 2024-08-20T22:39:39.0343497Z mypy-extensions==1.0.0 2024-08-20T22:39:39.0343814Z networkx==2.8.8 2024-08-20T22:39:39.0344100Z ninja==1.10.2 2024-08-20T22:39:39.0344590Z nose==1.3.7 2024-08-20T22:39:39.0344872Z numba==0.55.2 2024-08-20T22:39:39.0345149Z numpy==1.21.2 2024-08-20T22:39:39.0345503Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:39:39.0345927Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:39:39.0346366Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:39:39.0346815Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:39:39.0347263Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:39:39.0347667Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:39:39.0348067Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:39:39.0348492Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:39:39.0348932Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:39:39.0349357Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:39:39.0349749Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:39:39.0350164Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:39:39.0350501Z oauthlib==3.2.2 2024-08-20T22:39:39.0350788Z omegaconf==2.3.0 2024-08-20T22:39:39.0351078Z onnx==1.16.1 2024-08-20T22:39:39.0351375Z onnxscript==0.1.0.dev20240817 2024-08-20T22:39:39.0351724Z opacus==1.5.2 2024-08-20T22:39:39.0352056Z opencv-python==4.8.0.74 2024-08-20T22:39:39.0352419Z opt-einsum==3.3.0 2024-08-20T22:39:39.0352721Z optree==0.12.1 2024-08-20T22:39:39.0353006Z packaging==24.1 2024-08-20T22:39:39.0353307Z pandas==2.0.3 2024-08-20T22:39:39.0353588Z patch==1.16 2024-08-20T22:39:39.0353864Z pathspec==0.12.1 2024-08-20T22:39:39.0354162Z phonemizer==3.3.0 2024-08-20T22:39:39.0354459Z pillow==10.3.0 2024-08-20T22:39:39.0354752Z platformdirs==4.2.2 2024-08-20T22:39:39.0355100Z pluggy==1.5.0 2024-08-20T22:39:39.0355461Z pooch==1.8.2 2024-08-20T22:39:39.0355818Z portalocker==2.10.1 2024-08-20T22:39:39.0356127Z preshed==3.0.9 2024-08-20T22:39:39.0356415Z prettytable==3.11.0 2024-08-20T22:39:39.0356719Z protobuf==3.20.2 2024-08-20T22:39:39.0357013Z psutil==6.0.0 2024-08-20T22:39:39.0357358Z py-cpuinfo==9.0.0 2024-08-20T22:39:39.0357662Z pyaml==24.7.0 2024-08-20T22:39:39.0357942Z pyarrow==17.0.0 2024-08-20T22:39:39.0358231Z pyasn1==0.6.0 2024-08-20T22:39:39.0358517Z pyasn1_modules==0.4.0 2024-08-20T22:39:39.0359036Z pyclipper==1.3.0.post5 2024-08-20T22:39:39.0359364Z pycocotools==2.0.8 2024-08-20T22:39:39.0359666Z pycparser==2.22 2024-08-20T22:39:39.0359974Z pycryptodomex==3.20.0 2024-08-20T22:39:39.0360293Z pydantic==2.8.2 2024-08-20T22:39:39.0360601Z pydantic_core==2.20.1 2024-08-20T22:39:39.0360957Z pydot==3.0.1 2024-08-20T22:39:39.0361335Z pygame==2.6.0 2024-08-20T22:39:39.0361626Z PyGithub==2.3.0 2024-08-20T22:39:39.0361914Z Pygments==2.15.0 2024-08-20T22:39:39.0362211Z PyJWT==2.9.0 2024-08-20T22:39:39.0362496Z pylatexenc==2.10 2024-08-20T22:39:39.0362798Z pylint==3.2.6 2024-08-20T22:39:39.0363072Z PyNaCl==1.5.0 2024-08-20T22:39:39.0363355Z pynvml==11.5.3 2024-08-20T22:39:39.0363647Z pyparsing==3.1.2 2024-08-20T22:39:39.0363950Z pypdfium2==4.30.0 2024-08-20T22:39:39.0364240Z pysbd==0.3.4 2024-08-20T22:39:39.0364522Z PySocks==1.7.1 2024-08-20T22:39:39.0364810Z pytest==7.3.2 2024-08-20T22:39:39.0365165Z pytest-benchmark==4.0.0 2024-08-20T22:39:39.0365536Z pytest-cpp==2.3.0 2024-08-20T22:39:39.0365969Z pytest-flakefinder==1.1.0 2024-08-20T22:39:39.0366378Z pytest-rerunfailures==14.0 2024-08-20T22:39:39.0366768Z pytest-xdist==3.3.1 2024-08-20T22:39:39.0367134Z python-dateutil==2.9.0.post0 2024-08-20T22:39:39.0367926Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:39:39.0369337Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:39:39.0370288Z pytz==2024.1 2024-08-20T22:39:39.0370567Z PyWavelets==1.4.1 2024-08-20T22:39:39.0370939Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:39:39.0371385Z rapidfuzz==3.9.6 2024-08-20T22:39:39.0371674Z rdflib==7.0.0 2024-08-20T22:39:39.0372401Z redis==5.0.8 2024-08-20T22:39:39.0372704Z referencing==0.35.1 2024-08-20T22:39:39.0373010Z regex==2024.7.24 2024-08-20T22:39:39.0373300Z requests==2.32.3 2024-08-20T22:39:39.0373666Z requests-oauthlib==2.0.0 2024-08-20T22:39:39.0374006Z resampy==0.4.3 2024-08-20T22:39:39.0374293Z rfc3986==1.5.0 2024-08-20T22:39:39.0374579Z rich==13.7.1 2024-08-20T22:39:39.0374877Z rockset==1.0.3 2024-08-20T22:39:39.0375318Z rpds-py==0.20.0 2024-08-20T22:39:39.0375608Z rsa==4.9 2024-08-20T22:39:39.0375878Z s3transfer==0.5.2 2024-08-20T22:39:39.0376188Z safetensors==0.4.4 2024-08-20T22:39:39.0376540Z scikit-image==0.21.0 2024-08-20T22:39:39.0376892Z scikit-learn==1.5.1 2024-08-20T22:39:39.0377199Z scipy==1.10.1 2024-08-20T22:39:39.0377481Z segments==2.2.1 2024-08-20T22:39:39.0377770Z sentencepiece==0.2.0 2024-08-20T22:39:39.0378125Z sentry-sdk==2.13.0 2024-08-20T22:39:39.0378432Z setproctitle==1.3.3 2024-08-20T22:39:39.0378740Z shapely==2.0.6 2024-08-20T22:39:39.0379025Z shellingham==1.5.4 2024-08-20T22:39:39.0379333Z simplejson==3.19.3 2024-08-20T22:39:39.0379726Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:39:39.0380225Z smart-open==7.0.4 2024-08-20T22:39:39.0380525Z smmap==5.0.1 2024-08-20T22:39:39.0380817Z sortedcontainers==2.4.0 2024-08-20T22:39:39.0381157Z soundfile==0.12.1 2024-08-20T22:39:39.0381449Z soupsieve==2.6 2024-08-20T22:39:39.0381738Z soxr==0.4.0 2024-08-20T22:39:39.0382007Z spacy==3.7.6 2024-08-20T22:39:39.0382322Z spacy-legacy==3.0.12 2024-08-20T22:39:39.0382679Z spacy-loggers==1.0.5 2024-08-20T22:39:39.0382990Z srsly==2.4.8 2024-08-20T22:39:39.0383271Z stempeg==0.2.3 2024-08-20T22:39:39.0383559Z submitit==1.5.1 2024-08-20T22:39:39.0383846Z sympy==1.13.1 2024-08-20T22:39:39.0384128Z tabulate==0.9.0 2024-08-20T22:39:39.0384621Z tb-nightly==2.13.0a20230426 2024-08-20T22:39:39.0384977Z tensorboard==2.13.0 2024-08-20T22:39:39.0385354Z tensorboard-data-server==0.7.2 2024-08-20T22:39:39.0385728Z tensorboardX==2.6.2.2 2024-08-20T22:39:39.0386042Z termcolor==2.4.0 2024-08-20T22:39:39.0386333Z thinc==8.2.5 2024-08-20T22:39:39.0386620Z threadpoolctl==3.5.0 2024-08-20T22:39:39.0386927Z tifffile==2024.8.10 2024-08-20T22:39:39.0387731Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:39:39.0388892Z tlparse==0.3.25 2024-08-20T22:39:39.0389188Z tokenizers==0.15.2 2024-08-20T22:39:39.0389479Z tomli==2.0.1 2024-08-20T22:39:39.0389759Z tomlkit==0.13.2 2024-08-20T22:39:39.0391092Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b444e33ace8c516878aca2862d707788a4c01d8c6190fb13e010fb9a131a57c5 2024-08-20T22:39:39.0392919Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:39:39.0393979Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:39:39.0394949Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:39:39.0395934Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:39:39.0397025Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:39:39.0398122Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:39:39.0399188Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:39:39.0399887Z tornado==6.4.1 2024-08-20T22:39:39.0400175Z tqdm==4.66.5 2024-08-20T22:39:39.0400466Z transformers==4.38.1 2024-08-20T22:39:39.0400805Z treetable==0.2.5 2024-08-20T22:39:39.0401641Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:39:39.0402422Z typeguard==4.3.0 2024-08-20T22:39:39.0402711Z typer==0.12.4 2024-08-20T22:39:39.0403012Z typing_extensions==4.12.2 2024-08-20T22:39:39.0403348Z tzdata==2024.1 2024-08-20T22:39:39.0403638Z Unidecode==1.3.8 2024-08-20T22:39:39.0404000Z unittest-xml-reporting==3.2.0 2024-08-20T22:39:39.0404377Z uritemplate==4.1.1 2024-08-20T22:39:39.0404680Z urllib3==1.26.19 2024-08-20T22:39:39.0404977Z visdom==0.2.4 2024-08-20T22:39:39.0405257Z wandb==0.17.7 2024-08-20T22:39:39.0405537Z wasabi==1.1.3 2024-08-20T22:39:39.0405822Z wcwidth==0.2.13 2024-08-20T22:39:39.0406105Z weasel==0.4.1 2024-08-20T22:39:39.0406435Z websocket-client==1.8.0 2024-08-20T22:39:39.0406768Z Werkzeug==3.0.3 2024-08-20T22:39:39.0407055Z wrapt==1.16.0 2024-08-20T22:39:39.0407334Z xdoctest==1.1.0 2024-08-20T22:39:39.0407625Z xxhash==3.5.0 2024-08-20T22:39:39.0407914Z xyzservices==2024.6.0 2024-08-20T22:39:39.0408220Z yacs==0.1.8 2024-08-20T22:39:39.0408492Z yarl==1.9.4 2024-08-20T22:39:39.0408803Z z3-solver==4.12.2.0 2024-08-20T22:39:39.0409105Z zipp==3.20.0 2024-08-20T22:39:39.0598817Z + popd 2024-08-20T22:39:39.0599157Z ~/workspace 2024-08-20T22:39:39.0599518Z + [[ dynamic_inductor_torchbench != *cpu* ]] 2024-08-20T22:39:39.0599973Z + install_torchrec_and_fbgemm 2024-08-20T22:39:39.0600346Z + local torchrec_commit 2024-08-20T22:39:39.0602750Z ++ get_pinned_commit torchrec 2024-08-20T22:39:39.0603162Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:39:39.0641201Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:39:39.0641730Z + local fbgemm_commit 2024-08-20T22:39:39.0644490Z ++ get_pinned_commit fbgemm 2024-08-20T22:39:39.0644884Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:39:39.0688202Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:39.0688960Z + pip_uninstall torchrec-nightly 2024-08-20T22:39:39.0689431Z + pip uninstall -y torchrec-nightly 2024-08-20T22:39:39.4443908Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:39:39.4809501Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:39:39.4810077Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:39:39.8490875Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:39:39.8846197Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:39.8847250Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:40.3933746Z Collecting setuptools-git-versioning 2024-08-20T22:39:40.4109329Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:39:40.5379328Z Collecting scikit-build 2024-08-20T22:39:40.5418471Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:39:40.6573938Z Collecting pyre-extensions 2024-08-20T22:39:40.6600675Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:39:40.7111533Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:39:40.7113332Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:39:40.7677436Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:39:40.7706595Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:39:40.8182766Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:39:40.8188519Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:39:40.8192708Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:39:40.8727354Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:39:40.8753848Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:39:40.9247008Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:39:40.9329481Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:39:40.9362913Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:39:40.9841887Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:39:41.0373793Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:39:41.0907480Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:39:41.1435627Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:39:42.0323591Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:39:42.3896757Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:39:42.5011160Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:39:42.5012377Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:42.5014255Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:42.8736226Z Collecting fbgemm-gpu 2024-08-20T22:39:42.8739039Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-r5f9sx4p/fbgemm-gpu_1df4369c617449ab84f26ab2a62cc107 2024-08-20T22:39:42.8769779Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-r5f9sx4p/fbgemm-gpu_1df4369c617449ab84f26ab2a62cc107 2024-08-20T22:39:43.9807488Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:39:43.9838462Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:44.4845771Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:44.9857554Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:44.9859532Z Running command git submodule update --init --recursive -q 2024-08-20T22:39:54.7570010Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:39:54.7590217Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:39:54.7597068Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:24:51.7004610Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T23:24:51.8843658Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252620178 sha256=e9c197f0be010dfedb6ae9a0cbafab73877bd4c1b1ebcfb162b02789e9a91c5b 2024-08-20T23:24:51.8863323Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:24:51.8864441Z Successfully built fbgemm-gpu 2024-08-20T23:24:52.7485090Z Installing collected packages: fbgemm-gpu 2024-08-20T23:24:56.2849690Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:24:57.1920757Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:57.1922295Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:57.5611017Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:57.5615802Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-4jn5khqa 2024-08-20T23:24:57.5643738Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-4jn5khqa 2024-08-20T23:24:58.3356216Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:24:58.3383257Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:58.6502145Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:58.9925447Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:59.2643993Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:24:59.2679772Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:24:59.2682755Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:24:59.2686034Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:24:59.3190826Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:24:59.3371818Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:24:59.3786596Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:24:59.3789452Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:24:59.3799260Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:24:59.3809500Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:24:59.4079376Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:24:59.4082762Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:24:59.4086460Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:24:59.4108861Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:24:59.4112786Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:24:59.4264426Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:24:59.4290199Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:24:59.4399890Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:24:59.4403346Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:24:59.4424298Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:24:59.4458116Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:24:59.4462238Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:24:59.4466094Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:24:59.4469207Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:24:59.4474385Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:24:59.4491323Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:24:59.4906829Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:24:59.5044392Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:24:59.5247330Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:24:59.5358730Z Building wheels for collected packages: torchrec 2024-08-20T23:25:00.0746775Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:25:00.0755127Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=9486a52d5ca887e7bec632e479b8f0f565bd2b02fe63e8b7844617ef0e5fbe12 2024-08-20T23:25:00.0756759Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:25:00.0780197Z Successfully built torchrec 2024-08-20T23:25:00.8770049Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:25:01.4902933Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:25:01.6286439Z ++ pwd 2024-08-20T23:25:01.6289211Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:25:01.6289754Z + test_dynamo_benchmark torchbench 1 2024-08-20T23:25:01.6293969Z ++ pwd 2024-08-20T23:25:01.6298072Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:01.6298745Z + local suite=torchbench 2024-08-20T23:25:01.6299094Z + shift 2024-08-20T23:25:01.6300300Z + local shard_id=1 2024-08-20T23:25:01.6300690Z + shift 2024-08-20T23:25:01.6303319Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:01.6304084Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-08-20T23:25:01.6304856Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T23:25:01.6305366Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:01.6306330Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-08-20T23:25:01.6307090Z ++ pwd 2024-08-20T23:25:01.6307584Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:01.6308266Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:01.6353159Z + local name=inference 2024-08-20T23:25:01.6353647Z + shift 2024-08-20T23:25:01.6353986Z + local suite=torchbench 2024-08-20T23:25:01.6354315Z + shift 2024-08-20T23:25:01.6354593Z + local shard_id=1 2024-08-20T23:25:01.6354897Z + shift 2024-08-20T23:25:01.6359618Z + partition_flags=() 2024-08-20T23:25:01.6359971Z + local partition_flags 2024-08-20T23:25:01.6360466Z + [[ -n 2 ]] 2024-08-20T23:25:01.6360844Z + [[ -n 1 ]] 2024-08-20T23:25:01.6361445Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:25:01.6362159Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:01.6362683Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-08-20T23:25:01.6363197Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:01.6363704Z + [[ dynamic_inductor_torchbench == *_avx2* ]] 2024-08-20T23:25:01.6364220Z + [[ dynamic_inductor_torchbench == *_avx512* ]] 2024-08-20T23:25:01.6366057Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:25:07.6105962Z 2024-08-20T23:25:08.5207881Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:08.5208500Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:08.5242085Z cuda eval llama 2024-08-20T23:26:03.8191248Z pass 2024-08-20T23:26:03.8437069Z TIMING: code_gen:5.23065 inductor_compile:18.98452 backend_compile:45.17305 entire_frame_compile:53.22646 total_wall_time:53.22646 2024-08-20T23:26:03.8439098Z STATS: call_* op count: 1058 | FakeTensorMode.__torch_dispatch__:36913 | attempt fast:1244 | fast is_contiguous:1244 | FakeTensor.__torch_dispatch__:2782 | ProxyTorchDispatchMode.__torch_dispatch__:10235 2024-08-20T23:26:03.8440905Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:09.6441674Z 2024-08-20T23:26:09.9480275Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:09.9480852Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:09.9481300Z cuda eval llama_v2_7b_16h 2024-08-20T23:26:09.9504502Z Traceback (most recent call last): 2024-08-20T23:26:09.9505277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:26:09.9506097Z ) = runner.load_model( 2024-08-20T23:26:09.9506995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:26:09.9507983Z benchmark = benchmark_cls( 2024-08-20T23:26:09.9508963Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:26:09.9509941Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:26:09.9510940Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-08-20T23:26:09.9512435Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:09.9514014Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:26:09.9515196Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:09.9516164Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:26:09.9516909Z self._skip_by_device_name() 2024-08-20T23:26:09.9517711Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:26:09.9518941Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:26:09.9520069Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-08-20T23:26:09.9520676Z 2024-08-20T23:26:09.9520801Z model_fail_to_load 2024-08-20T23:26:13.4110363Z 2024-08-20T23:26:13.7144678Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:13.7145828Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:13.7146832Z cuda eval llava 2024-08-20T23:26:13.7151272Z Traceback (most recent call last): 2024-08-20T23:26:13.7152153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:26:13.7153004Z ) = runner.load_model( 2024-08-20T23:26:13.7153721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:26:13.7154679Z benchmark = benchmark_cls( 2024-08-20T23:26:13.7155598Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:26:13.7156547Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:26:13.7157417Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:26:13.7158509Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:13.7159759Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:26:13.7160919Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:13.7161887Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:26:13.7162631Z self._skip_by_device_name() 2024-08-20T23:26:13.7163424Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:26:13.7164626Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:26:13.7165988Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:26:13.7166554Z 2024-08-20T23:26:13.7166671Z model_fail_to_load 2024-08-20T23:26:17.1713558Z 2024-08-20T23:26:17.9464408Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:17.9465447Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:17.9470410Z cuda eval maml 2024-08-20T23:26:17.9473144Z pass_due_to_skip 2024-08-20T23:26:17.9481223Z TIMING: 2024-08-20T23:26:17.9481611Z STATS: call_* op count: 0 2024-08-20T23:26:17.9482326Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:21.5052233Z 2024-08-20T23:26:22.0447479Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:22.0448084Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:22.0454190Z cuda eval maml_omniglot 2024-08-20T23:26:26.2281777Z pass 2024-08-20T23:26:26.2283192Z TIMING: code_gen:0.95028 inductor_compile:1.50615 backend_compile:2.64883 entire_frame_compile:3.05053 total_wall_time:3.05053 2024-08-20T23:26:26.2285332Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1958 | ProxyTorchDispatchMode.__torch_dispatch__:543 | FakeTensor.__torch_dispatch__:176 2024-08-20T23:26:26.2286451Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:30.0215822Z 2024-08-20T23:26:30.2379289Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-08-20T23:26:30.2555975Z 2024-08-20T23:26:30.2556176Z 2024-08-20T23:26:30.3003662Z 0% 0.00/16.9M [00:00 will be ignored 2024-08-20T23:28:02.3899554Z [rank0]:W0820 23:28:02.388000 33724 torch/_dynamo/backends/distributed.py:89] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:28:02.4136767Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:28:02.4144359Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:28:02.4146043Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:28:02.4147517Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:28:02.4148915Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ 0 │ 470 │ 44910720 │ 2024-08-20T23:28:02.4150375Z [rank0]:W0820 23:28:02.413000 33724 torch/_dynamo/backends/distributed.py:106] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:28:23.7976012Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 2024-08-20T23:28:23.7977263Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward 2024-08-20T23:28:23.7978477Z self._momentum_update_key_encoder() # update the key encoder 2024-08-20T23:28:23.7980070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:28:23.7981088Z return func(*args, **kwargs) 2024-08-20T23:28:23.7982055Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder 2024-08-20T23:28:23.7983174Z param_k.mul_(self.m).add_(param_q.mul(1. - self.m)) 2024-08-20T23:28:23.7983540Z 2024-08-20T23:28:23.9165186Z [rank0]:W0820 23:28:23.915000 33724 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-20T23:28:23.9271014Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-08-20T23:28:23.9273075Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-08-20T23:28:23.9274086Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-08-20T23:28:23.9274459Z 2024-08-20T23:28:26.8588477Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:28:26.8589941Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:28:26.8591015Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] or: 2024-08-20T23:28:26.8592071Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:28:26.8593426Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] to include these operations in the captured graph. 2024-08-20T23:28:26.8594483Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:26.8595435Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break: from user code at: 2024-08-20T23:28:26.8597180Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:28:26.8598906Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] self._dequeue_and_enqueue(k) 2024-08-20T23:28:26.8600852Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:28:26.8602371Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] return func(*args, **kwargs) 2024-08-20T23:28:26.8603985Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:28:26.8605600Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] ptr = int(self.queue_ptr) 2024-08-20T23:28:26.8606530Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:26.8607312Z [rank0]:W0820 23:28:26.857000 33724 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:43.0328594Z pass 2024-08-20T23:28:43.0338389Z TIMING: entire_frame_compile:49.03426 code_gen:9.37881 inductor_compile:17.28767 backend_compile:29.0466 total_wall_time:49.03426 2024-08-20T23:28:43.0339900Z STATS: call_* op count: 879 | FakeTensorMode.__torch_dispatch__:39553 | FakeTensor.__torch_dispatch__:7974 | ProxyTorchDispatchMode.__torch_dispatch__:11976 2024-08-20T23:28:43.0341022Z Dynamo produced 7 graphs covering 879 ops with 5 graph breaks (3 unique) 2024-08-20T23:28:44.0518784Z [rank0]:[W820 23:28:44.993761546 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:28:48.7647720Z 2024-08-20T23:28:49.0659871Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:49.0660757Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:49.0661616Z cuda eval moondream 2024-08-20T23:28:49.0667283Z Traceback (most recent call last): 2024-08-20T23:28:49.0668053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:28:49.0669149Z ) = runner.load_model( 2024-08-20T23:28:49.0669829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:28:49.0670665Z benchmark = benchmark_cls( 2024-08-20T23:28:49.0671545Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:28:49.0672786Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:28:49.0674016Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-08-20T23:28:49.0675226Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:28:49.0676516Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:28:49.0677704Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:28:49.0678684Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:28:49.0679430Z self._skip_by_device_name() 2024-08-20T23:28:49.0680216Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:28:49.0681439Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:28:49.0682533Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-08-20T23:28:49.0683110Z 2024-08-20T23:28:49.0683238Z model_fail_to_load 2024-08-20T23:28:52.4756532Z 2024-08-20T23:28:54.4578091Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:28:54.7288028Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:28:54.7289212Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:28:54.7290147Z using fused AdamW: True 2024-08-20T23:28:55.0587032Z 2024-08-20T23:28:55.0587447Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:28:55.0622726Z cuda eval nanogpt 2024-08-20T23:29:31.7793767Z pass 2024-08-20T23:29:31.7812910Z TIMING: code_gen:2.61807 inductor_compile:8.45124 backend_compile:26.68169 entire_frame_compile:34.96197 total_wall_time:34.96197 2024-08-20T23:29:31.7814646Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:34279 | attempt fast:2346 | fast is_contiguous:2346 | FakeTensor.__torch_dispatch__:2052 | ProxyTorchDispatchMode.__torch_dispatch__:8279 2024-08-20T23:29:31.7815953Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:36.7421082Z 2024-08-20T23:29:39.1939688Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:39.1940163Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:29:39.1944122Z cuda eval nvidia_deeprecommender 2024-08-20T23:29:43.6953146Z pass 2024-08-20T23:29:43.6990164Z TIMING: code_gen:0.39074 inductor_compile:0.93217 backend_compile:2.92167 entire_frame_compile:3.25221 total_wall_time:3.25221 2024-08-20T23:29:43.6991561Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1366 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170 2024-08-20T23:29:43.6992678Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:47.4508802Z 2024-08-20T23:29:48.4294679Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:48.4295408Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:48.4314747Z cuda eval opacus_cifar10 2024-08-20T23:29:58.4515697Z pass 2024-08-20T23:29:58.4517046Z TIMING: code_gen:2.7777 inductor_compile:4.16243 backend_compile:6.43671 entire_frame_compile:8.662 total_wall_time:8.662 2024-08-20T23:29:58.4520421Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:5086 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866 2024-08-20T23:29:58.4521701Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:02.5202504Z 2024-08-20T23:30:03.1948899Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:03.1949370Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:03.2016709Z cuda eval phlippe_densenet 2024-08-20T23:30:23.7366943Z pass 2024-08-20T23:30:23.7368099Z TIMING: code_gen:4.55521 inductor_compile:7.84768 backend_compile:13.45373 entire_frame_compile:19.09532 total_wall_time:19.09532 2024-08-20T23:30:23.7369487Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:20818 | ProxyTorchDispatchMode.__torch_dispatch__:7380 | FakeTensor.__torch_dispatch__:2741 2024-08-20T23:30:28.1208954Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:28.1209476Z 2024-08-20T23:30:28.7056622Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:28.7057220Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:28.7080843Z cuda eval phlippe_resnet 2024-08-20T23:30:36.7985036Z pass 2024-08-20T23:30:36.7986457Z TIMING: code_gen:1.39478 inductor_compile:2.56912 backend_compile:4.97166 entire_frame_compile:6.85021 total_wall_time:6.85021 2024-08-20T23:30:36.7987909Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:7943 | ProxyTorchDispatchMode.__torch_dispatch__:2780 | FakeTensor.__torch_dispatch__:1018 2024-08-20T23:30:36.7989683Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:40.7908484Z 2024-08-20T23:30:41.1380920Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:41.1381526Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:41.1382126Z cuda eval pyhpc_equation_of_state 2024-08-20T23:30:47.7900498Z pass 2024-08-20T23:30:47.7901325Z TIMING: code_gen:1.64814 inductor_compile:3.33391 backend_compile:4.97445 entire_frame_compile:5.53605 total_wall_time:5.53605 2024-08-20T23:30:47.7902680Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8846 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061 2024-08-20T23:30:47.7903835Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:51.6431162Z 2024-08-20T23:30:51.9791585Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:51.9792065Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:51.9795472Z cuda eval pyhpc_isoneutral_mixing 2024-08-20T23:31:17.4935926Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 2024-08-20T23:31:17.4937196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2546, in run_n_iterations 2024-08-20T23:31:17.4938125Z return self.model_iter_fn(mod, inputs, collect_outputs=True) 2024-08-20T23:31:17.4939139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:31:17.4939867Z return mod(*inputs) 2024-08-20T23:31:17.4940734Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward 2024-08-20T23:31:17.4941760Z return isoneutral_pytorch.isoneutral_diffusion_pre( 2024-08-20T23:31:17.4943164Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre 2024-08-20T23:31:17.4945085Z K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :]) 2024-08-20T23:31:17.4945458Z 2024-08-20T23:31:17.5009392Z pass 2024-08-20T23:31:17.5010470Z TIMING: code_gen:5.29822 inductor_compile:14.59765 backend_compile:22.11924 entire_frame_compile:24.15782 total_wall_time:24.15782 2024-08-20T23:31:17.5012418Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:41429 | FakeTensor.__torch_dispatch__:5051 | ProxyTorchDispatchMode.__torch_dispatch__:14005 2024-08-20T23:31:17.5014048Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-08-20T23:31:22.3157753Z 2024-08-20T23:31:23.0937515Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:31:23.2148024Z 2024-08-20T23:31:23.2148388Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:23.2149150Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:31:23.2150249Z cuda eval pyhpc_turbulent_kinetic_energy 2024-08-20T23:31:23.2597949Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:31:51.4519504Z pass 2024-08-20T23:31:51.4575227Z TIMING: code_gen:10.01164 inductor_compile:16.1966 backend_compile:24.55946 entire_frame_compile:26.81066 total_wall_time:26.81066 2024-08-20T23:31:51.4577182Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:38624 | FakeTensor.__torch_dispatch__:2563 | ProxyTorchDispatchMode.__torch_dispatch__:14635 2024-08-20T23:31:51.4578820Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-08-20T23:31:56.1886056Z 2024-08-20T23:31:57.0227781Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:57.0228287Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:57.0239976Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-08-20T23:32:14.2978827Z pass 2024-08-20T23:32:14.2986485Z TIMING: code_gen:3.03235 inductor_compile:5.14992 backend_compile:7.15948 entire_frame_compile:8.93757 total_wall_time:8.93757 2024-08-20T23:32:14.2988157Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:8225 | ProxyTorchDispatchMode.__torch_dispatch__:3743 | FakeTensor.__torch_dispatch__:1038 2024-08-20T23:32:14.2989290Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-08-20T23:32:18.4215522Z 2024-08-20T23:32:19.5774709Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:32:19.5775325Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:32:19.5797864Z cuda eval pytorch_stargan 2024-08-20T23:32:44.3474252Z pass 2024-08-20T23:32:44.3484282Z TIMING: code_gen:2.34731 inductor_compile:4.5281 backend_compile:7.22448 entire_frame_compile:9.01208 total_wall_time:9.01208 2024-08-20T23:32:44.3485695Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:12390 | FakeTensor.__torch_dispatch__:1212 | ProxyTorchDispatchMode.__torch_dispatch__:4591 2024-08-20T23:32:44.3486799Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-08-20T23:32:48.5041388Z 2024-08-20T23:32:49.2210433Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:32:49.2210927Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:32:49.2232231Z cuda eval pytorch_unet 2024-08-20T23:33:11.9926061Z pass 2024-08-20T23:33:11.9951668Z TIMING: code_gen:4.1752 inductor_compile:6.31953 backend_compile:8.67001 entire_frame_compile:10.81177 total_wall_time:10.81177 2024-08-20T23:33:11.9953564Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:11341 | ProxyTorchDispatchMode.__torch_dispatch__:4337 | FakeTensor.__torch_dispatch__:1309 2024-08-20T23:33:11.9955063Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:33:16.1031852Z 2024-08-20T23:33:16.9132647Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-08-20T23:33:16.9306006Z 2024-08-20T23:33:16.9306193Z 2024-08-20T23:33:17.0317864Z 0% 0.00/230M [00:00 2024-08-20T23:42:17.9723974Z torchbench_main() 2024-08-20T23:42:17.9724785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torchbench_main 2024-08-20T23:42:17.9725635Z main(TorchBenchmarkRunner(), original_dir) 2024-08-20T23:42:17.9726385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4087, in main 2024-08-20T23:42:17.9729061Z process_entry(0, runner, original_dir, args) 2024-08-20T23:42:17.9729930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4019, in process_entry 2024-08-20T23:42:17.9733735Z return run(runner, args, original_dir) 2024-08-20T23:42:17.9734504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4661, in run 2024-08-20T23:42:17.9739407Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-08-20T23:42:17.9740197Z AssertionError: nothing in example_inputs had a dim with 4 2024-08-20T23:42:21.5355702Z Run failed with return code: 1 2024-08-20T23:42:21.5356289Z Output: None 2024-08-20T23:42:21.5356683Z Error: None 2024-08-20T23:42:24.3283627Z 2024-08-20T23:42:24.5088875Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-08-20T23:42:24.5266216Z 2024-08-20T23:42:24.5266480Z 2024-08-20T23:42:24.5498765Z 0% 0.00/8.79M [00:00 2024-08-20T23:48:29.1019738Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-20T23:48:29.1020325Z 2024-08-20T23:48:30.2639137Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:48:30.2641707Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:48:30.2643843Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] or: 2024-08-20T23:48:30.2645855Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:48:30.2647051Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] to include these operations in the captured graph. 2024-08-20T23:48:30.2648042Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:48:30.2648938Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] Graph break: from user code at: 2024-08-20T23:48:30.2650987Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-20T23:48:30.2653476Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-20T23:48:30.2655472Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-20T23:48:30.2657160Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-20T23:48:30.2658908Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-20T23:48:30.2660418Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] self.scales, self.map_levels = _setup_scales( 2024-08-20T23:48:30.2662104Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-20T23:48:30.2663746Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:48:30.2665653Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-20T23:48:30.2667276Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:48:30.2669053Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:48:30.2670631Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:48:30.2671650Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:48:30.2672900Z W0820 23:48:30.263000 43908 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:49:12.3329390Z pass 2024-08-20T23:49:12.3342364Z TIMING: entire_frame_compile:69.34717 code_gen:13.08965 inductor_compile:22.56821 backend_compile:33.70608 total_wall_time:69.34717 2024-08-20T23:49:12.3344260Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:61102 | FakeTensor.__torch_dispatch__:6028 | ProxyTorchDispatchMode.__torch_dispatch__:18373 | attempt fast:152 | fast is_contiguous:96 | slow both tensors nontrivially broadcast:56 2024-08-20T23:49:12.3345912Z Dynamo produced 25 graphs covering 1358 ops with 16 graph breaks (2 unique) 2024-08-20T23:49:17.5542665Z 2024-08-20T23:49:20.0417962Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:20.0418449Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:49:20.0510213Z cuda eval yolov3 2024-08-20T23:49:45.1537961Z W0820 23:49:45.153000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:45.4062262Z W0820 23:49:45.405000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:45.6546917Z W0820 23:49:45.653000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:45.9627418Z W0820 23:49:45.962000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:45.9633583Z W0820 23:49:45.962000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:45.9641647Z W0820 23:49:45.963000 44344 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:49:55.7114065Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg298_1). Found from : 2024-08-20T23:49:55.7115191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T23:49:55.7117771Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T23:49:55.7119005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:49:55.7119831Z return mod(*inputs) 2024-08-20T23:49:55.7120742Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-20T23:49:55.7121786Z return self.forward_once(x) 2024-08-20T23:49:55.7122734Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-08-20T23:49:55.7123627Z yolo_out.append(module(x, out)) 2024-08-20T23:49:55.7124504Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-08-20T23:49:55.7125594Z self.create_grids((nx, ny), p.device) 2024-08-20T23:49:55.7126791Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-08-20T23:49:55.7127739Z self.anchor_vec = self.anchor_vec.to(device) 2024-08-20T23:49:55.7128079Z 2024-08-20T23:49:58.0560126Z pass 2024-08-20T23:49:58.0675362Z TIMING: code_gen:9.48264 inductor_compile:14.86791 backend_compile:23.02329 entire_frame_compile:30.43559 total_wall_time:30.43559 2024-08-20T23:49:58.0676874Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:31336 | FakeTensor.__torch_dispatch__:3524 | ProxyTorchDispatchMode.__torch_dispatch__:12245 2024-08-20T23:49:58.0677994Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique) 2024-08-20T23:50:00.1426080Z accuracy pass_rate=82.22% 2024-08-20T23:50:00.1429458Z calls_captured gmean=0.00x mean=526.911x 2024-08-20T23:50:00.1433116Z unique_graphs gmean=0.00x mean=1.667x 2024-08-20T23:50:00.1436990Z graph_breaks gmean=0.00x mean=0.733x 2024-08-20T23:50:00.1440484Z unique_graph_breaks gmean=0.00x mean=0.178x 2024-08-20T23:50:00.1444332Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:50:00.1448090Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:50:00.1451711Z cudagraph_skips gmean=0.00x mean=0.244x 2024-08-20T23:50:01.2799443Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_inference.csv 2024-08-20T23:50:01.6246462Z llama PASS 2024-08-20T23:50:01.6250227Z llama_v2_7b_16h XFAIL 2024-08-20T23:50:01.6254619Z llava XFAIL 2024-08-20T23:50:01.6258985Z maml XFAIL 2024-08-20T23:50:01.6263441Z maml_omniglot PASS 2024-08-20T23:50:01.6268047Z mnasnet1_0 PASS 2024-08-20T23:50:01.6272657Z mobilenet_v2 PASS 2024-08-20T23:50:01.6277159Z mobilenet_v2_quantized_qat XFAIL 2024-08-20T23:50:01.6281430Z mobilenet_v3_large PASS 2024-08-20T23:50:01.6285991Z moco PASS 2024-08-20T23:50:01.6290333Z moondream XFAIL 2024-08-20T23:50:01.6294741Z nanogpt PASS 2024-08-20T23:50:01.6299165Z nvidia_deeprecommender PASS 2024-08-20T23:50:01.6303444Z opacus_cifar10 PASS 2024-08-20T23:50:01.6308115Z phlippe_densenet PASS 2024-08-20T23:50:01.6312560Z phlippe_resnet PASS 2024-08-20T23:50:01.6316935Z pyhpc_equation_of_state PASS 2024-08-20T23:50:01.6321261Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:50:01.6325837Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:50:01.6330143Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:50:01.6334449Z pytorch_stargan PASS 2024-08-20T23:50:01.6338799Z pytorch_unet PASS 2024-08-20T23:50:01.6343159Z resnet152 PASS 2024-08-20T23:50:01.6347759Z resnet18 PASS 2024-08-20T23:50:01.6352224Z resnet50 PASS 2024-08-20T23:50:01.6356471Z resnet50_quantized_qat XFAIL 2024-08-20T23:50:01.6360841Z resnext50_32x4d PASS 2024-08-20T23:50:01.6365291Z sam PASS 2024-08-20T23:50:01.6369648Z shufflenet_v2_x1_0 PASS 2024-08-20T23:50:01.6374529Z soft_actor_critic PASS 2024-08-20T23:50:01.6378871Z speech_transformer PASS 2024-08-20T23:50:01.6383199Z squeezenet1_1 PASS 2024-08-20T23:50:01.6387827Z stable_diffusion_text_encoder PASS 2024-08-20T23:50:01.6392259Z stable_diffusion_unet XFAIL 2024-08-20T23:50:01.6396452Z timm_efficientnet PASS 2024-08-20T23:50:01.6400806Z timm_regnet PASS 2024-08-20T23:50:01.6405132Z timm_resnest PASS 2024-08-20T23:50:01.6409421Z timm_vision_transformer PASS 2024-08-20T23:50:01.6413877Z timm_vision_transformer_large XFAIL 2024-08-20T23:50:01.6418183Z timm_vovnet PASS 2024-08-20T23:50:01.6422671Z torch_multimodal_clip PASS 2024-08-20T23:50:01.6427149Z tts_angular PASS 2024-08-20T23:50:01.6431531Z vgg16 PASS 2024-08-20T23:50:01.6435827Z vision_maskrcnn PASS 2024-08-20T23:50:01.6440238Z yolov3 PASS 2024-08-20T23:50:01.6956985Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_inference.csv 2024-08-20T23:50:02.0308929Z llama PASS 2024-08-20T23:50:02.0313584Z llama_v2_7b_16h PASS 2024-08-20T23:50:02.0317364Z llava PASS 2024-08-20T23:50:02.0321801Z maml PASS 2024-08-20T23:50:02.0326219Z maml_omniglot PASS 2024-08-20T23:50:02.0330683Z mnasnet1_0 PASS 2024-08-20T23:50:02.0335167Z mobilenet_v2 PASS 2024-08-20T23:50:02.0339409Z mobilenet_v2_quantized_qat PASS 2024-08-20T23:50:02.0344080Z mobilenet_v3_large PASS 2024-08-20T23:50:02.0348522Z moco PASS 2024-08-20T23:50:02.0352964Z moondream PASS 2024-08-20T23:50:02.0357248Z nanogpt PASS 2024-08-20T23:50:02.0361498Z nvidia_deeprecommender PASS 2024-08-20T23:50:02.0365846Z opacus_cifar10 PASS 2024-08-20T23:50:02.0370159Z phlippe_densenet PASS 2024-08-20T23:50:02.0375127Z phlippe_resnet PASS 2024-08-20T23:50:02.0379480Z pyhpc_equation_of_state PASS 2024-08-20T23:50:02.0383903Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:50:02.0388603Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:50:02.0392924Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:50:02.0396906Z pytorch_stargan PASS 2024-08-20T23:50:02.0401287Z pytorch_unet PASS 2024-08-20T23:50:02.0405786Z resnet152 PASS 2024-08-20T23:50:02.0410026Z resnet18 PASS 2024-08-20T23:50:02.0414402Z resnet50 PASS 2024-08-20T23:50:02.0418706Z resnet50_quantized_qat PASS 2024-08-20T23:50:02.0423111Z resnext50_32x4d PASS 2024-08-20T23:50:02.0427858Z sam PASS 2024-08-20T23:50:02.0432170Z shufflenet_v2_x1_0 PASS 2024-08-20T23:50:02.0436487Z soft_actor_critic PASS 2024-08-20T23:50:02.0440901Z speech_transformer PASS 2024-08-20T23:50:02.0445231Z squeezenet1_1 PASS 2024-08-20T23:50:02.0449645Z stable_diffusion_text_encoder PASS 2024-08-20T23:50:02.0453986Z stable_diffusion_unet PASS 2024-08-20T23:50:02.0458603Z timm_efficientnet PASS 2024-08-20T23:50:02.0462748Z timm_regnet PASS 2024-08-20T23:50:02.0467295Z timm_resnest PASS 2024-08-20T23:50:02.0471627Z timm_vision_transformer PASS 2024-08-20T23:50:02.0476498Z timm_vision_transformer_large PASS 2024-08-20T23:50:02.0480689Z timm_vovnet PASS 2024-08-20T23:50:02.0485088Z torch_multimodal_clip PASS 2024-08-20T23:50:02.0489462Z tts_angular PASS 2024-08-20T23:50:02.0493873Z vgg16 PASS 2024-08-20T23:50:02.0498211Z vision_maskrcnn PASS 2024-08-20T23:50:02.0502573Z yolov3 PASS 2024-08-20T23:50:02.1018098Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-08-20T23:50:02.1021110Z ++ pwd 2024-08-20T23:50:02.1024199Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:50:02.1025017Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:50:02.1048901Z + local name=training 2024-08-20T23:50:02.1049799Z + shift 2024-08-20T23:50:02.1050529Z + local suite=torchbench 2024-08-20T23:50:02.1051374Z + shift 2024-08-20T23:50:02.1052066Z + local shard_id=1 2024-08-20T23:50:02.1052793Z + shift 2024-08-20T23:50:02.1053115Z + partition_flags=() 2024-08-20T23:50:02.1053451Z + local partition_flags 2024-08-20T23:50:02.1053857Z + [[ -n 2 ]] 2024-08-20T23:50:02.1054169Z + [[ -n 1 ]] 2024-08-20T23:50:02.1054916Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:50:02.1055762Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:50:02.1056333Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-08-20T23:50:02.1056881Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:50:02.1057429Z + [[ dynamic_inductor_torchbench == *_avx2* ]] 2024-08-20T23:50:02.1057955Z + [[ dynamic_inductor_torchbench == *_avx512* ]] 2024-08-20T23:50:02.1059779Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-08-20T23:50:07.8351737Z 2024-08-20T23:50:08.1426365Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:08.1427488Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:08.1428518Z cuda train llava 2024-08-20T23:50:08.1434354Z Traceback (most recent call last): 2024-08-20T23:50:08.1435309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:50:08.1436071Z ) = runner.load_model( 2024-08-20T23:50:08.1436731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-20T23:50:08.1437455Z benchmark = benchmark_cls( 2024-08-20T23:50:08.1438176Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:50:08.1438951Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:50:08.1439811Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:50:08.1441183Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:50:08.1442824Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:50:08.1444011Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:50:08.1444987Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:50:08.1446065Z self._skip_by_device_name() 2024-08-20T23:50:08.1446868Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:50:08.1448271Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:50:08.1449338Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:50:08.1449908Z 2024-08-20T23:50:08.1450029Z model_fail_to_load 2024-08-20T23:50:11.5888108Z 2024-08-20T23:50:12.2470710Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:12.2471413Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:12.2472315Z cuda train maml_omniglot 2024-08-20T23:50:23.5431552Z W0820 23:50:23.542000 44792 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:50:29.1296470Z pass 2024-08-20T23:50:29.1297312Z TIMING: entire_frame_compile:14.94767 code_gen:7.47599 inductor_compile:9.54026 backend_compile:13.47252 total_wall_time:18.5755 2024-08-20T23:50:29.1300397Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:4981 | attempt fast:172 | fast is_contiguous:172 | ProxyTorchDispatchMode.__torch_dispatch__:1526 | FakeTensor.__torch_dispatch__:1147 2024-08-20T23:50:29.1301834Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-20T23:50:33.3116739Z 2024-08-20T23:50:35.8764935Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:35.8765950Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:50:35.8766807Z cuda train mnasnet1_0 2024-08-20T23:51:25.3612925Z W0820 23:51:25.360000 45131 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:52:06.5745946Z pass 2024-08-20T23:52:06.5774689Z TIMING: entire_frame_compile:83.9845 code_gen:26.11939 inductor_compile:44.63192 backend_compile:67.16285 total_wall_time:98.29408 2024-08-20T23:52:06.5776316Z STATS: call_* op count: 647 | FakeTensorMode.__torch_dispatch__:45886 | attempt fast:1436 | fast is_contiguous:1436 | ProxyTorchDispatchMode.__torch_dispatch__:17470 | FakeTensor.__torch_dispatch__:13506 2024-08-20T23:52:06.5777898Z Dynamo produced 3 graphs covering 647 ops with 7 graph breaks (5 unique) 2024-08-20T23:52:14.0115318Z 2024-08-20T23:52:16.0233156Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:16.0233681Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:52:16.0234125Z cuda train mobilenet_v2 2024-08-20T23:53:17.4413298Z pass 2024-08-20T23:53:17.4463155Z TIMING: entire_frame_compile:56.3078 code_gen:10.11342 inductor_compile:24.59073 backend_compile:43.53502 total_wall_time:71.26666 2024-08-20T23:53:17.4464947Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:35650 | attempt fast:1572 | fast is_contiguous:1572 | FakeTensor.__torch_dispatch__:6802 | ProxyTorchDispatchMode.__torch_dispatch__:10376 2024-08-20T23:53:17.4466284Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-08-20T23:53:23.9364780Z 2024-08-20T23:53:25.9118179Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:25.9118757Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:53:25.9120854Z cuda train mobilenet_v2_quantized_qat 2024-08-20T23:53:25.9122178Z Traceback (most recent call last): 2024-08-20T23:53:25.9123112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:53:25.9124106Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:53:25.9125072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:53:25.9127120Z pred = mod(*cloned_inputs) 2024-08-20T23:53:25.9128580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:53:25.9130140Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:53:25.9131324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:53:25.9132118Z raise e 2024-08-20T23:53:25.9133170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:53:25.9134325Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:53:25.9135735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:53:25.9136939Z return self._call_impl(*args, **kwargs) 2024-08-20T23:53:25.9138148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:53:25.9138976Z return forward_call(*args, **kwargs) 2024-08-20T23:53:25.9139446Z File ".3", line 207, in forward 2024-08-20T23:53:25.9140193Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-08-20T23:53:25.9141455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:53:25.9142336Z return self._call_impl(*args, **kwargs) 2024-08-20T23:53:25.9143262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:53:25.9144082Z return forward_call(*args, **kwargs) 2024-08-20T23:53:25.9145192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:53:25.9146088Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:53:25.9146630Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:53:25.9147006Z 2024-08-20T23:53:25.9147300Z The above exception was the direct cause of the following exception: 2024-08-20T23:53:25.9147748Z 2024-08-20T23:53:25.9147902Z Traceback (most recent call last): 2024-08-20T23:53:25.9148544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:53:25.9149186Z ) = runner.load_model( 2024-08-20T23:53:25.9149847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:53:25.9150604Z self.validate_model(model, example_inputs) 2024-08-20T23:53:25.9151348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:53:25.9152096Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:53:25.9152559Z RuntimeError: Eager run failed 2024-08-20T23:53:25.9152808Z 2024-08-20T23:53:25.9152943Z eager_fail_to_run 2024-08-20T23:53:29.5002810Z 2024-08-20T23:53:31.9342646Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:31.9343246Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:53:31.9343801Z cuda train mobilenet_v3_large 2024-08-20T23:54:41.3675185Z W0820 23:54:41.366000 46798 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:55:27.2389142Z E0820 23:55:27.238000 46798 torch/_dynamo/utils.py:1688] RMSE (res-fp64): 11.25649, (ref-fp64): 1.43235 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0 2024-08-20T23:55:27.2391074Z E0820 23:55:27.238000 46798 torch/_dynamo/utils.py:1580] Accuracy failed for key name features.6.block.0.1.weight.grad 2024-08-20T23:55:27.2413657Z pass 2024-08-20T23:55:27.2454076Z TIMING: entire_frame_compile:107.38338 code_gen:30.50725 inductor_compile:54.4908 backend_compile:84.92766 total_wall_time:126.45177 2024-08-20T23:55:27.2455738Z STATS: call_* op count: 730 | FakeTensorMode.__torch_dispatch__:57061 | attempt fast:2054 | fast is_contiguous:2054 | FakeTensor.__torch_dispatch__:16208 | ProxyTorchDispatchMode.__torch_dispatch__:19401 2024-08-20T23:55:27.2458693Z Dynamo produced 3 graphs covering 730 ops with 7 graph breaks (5 unique) 2024-08-20T23:55:36.0182605Z 2024-08-20T23:55:38.3793879Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:38.3794464Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:55:38.3795448Z cuda train moco 2024-08-20T23:55:43.7355656Z [rank0]:W0820 23:55:43.734000 47931 torch/_logging/_internal.py:1058] [3/0] Profiler function will be ignored 2024-08-20T23:55:54.1985225Z [rank0]:W0820 23:55:54.197000 47931 torch/_dynamo/backends/distributed.py:89] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:55:54.2227448Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:55:54.2232650Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:55:54.2234206Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:55:54.2235666Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:55:54.2237102Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ 0 │ 480 │ 94032128 │ 2024-08-20T23:55:54.2238551Z [rank0]:W0820 23:55:54.222000 47931 torch/_dynamo/backends/distributed.py:106] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:56:47.8559114Z skipping cudagraphs due to mutated inputs (161 instances) 2024-08-20T23:56:48.1687519Z [rank0]:W0820 23:56:48.168000 47931 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-20T23:56:48.1918954Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-08-20T23:56:48.1920163Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-08-20T23:56:48.1921148Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-08-20T23:56:48.1921514Z 2024-08-20T23:56:52.3759663Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:56:52.3762347Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:56:52.3763717Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] or: 2024-08-20T23:56:52.3764773Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:56:52.3766030Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] to include these operations in the captured graph. 2024-08-20T23:56:52.3767052Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:56:52.3768000Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break: from user code at: 2024-08-20T23:56:52.3769720Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:56:52.3771427Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] self._dequeue_and_enqueue(k) 2024-08-20T23:56:52.3774022Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:56:52.3775709Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] return func(*args, **kwargs) 2024-08-20T23:56:52.3777573Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:56:52.3779417Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] ptr = int(self.queue_ptr) 2024-08-20T23:56:52.3780449Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:56:52.3781297Z [rank0]:W0820 23:56:52.375000 47931 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:57:27.7930904Z pass 2024-08-20T23:57:27.8301010Z TIMING: entire_frame_compile:94.8674 inductor_compile:39.18586 backend_compile:70.75586 code_gen:19.69769 total_wall_time:110.54157 2024-08-20T23:57:27.8302798Z STATS: call_* op count: 891 | FakeTensorMode.__torch_dispatch__:66010 | ProxyTorchDispatchMode.__torch_dispatch__:20276 | attempt fast:2680 | fast is_contiguous:2664 | slow no contiguity match:16 | FakeTensor.__torch_dispatch__:10810 2024-08-20T23:57:27.8304704Z Dynamo produced 10 graphs covering 891 ops with 11 graph breaks (8 unique) 2024-08-20T23:57:30.2269658Z [rank0]:[W820 23:57:30.168876536 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:57:36.4435200Z 2024-08-20T23:57:38.4189583Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:57:38.6959598Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:57:38.6960913Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:57:38.6962015Z using fused AdamW: True 2024-08-20T23:57:39.1479331Z 2024-08-20T23:57:39.1480240Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:57:39.1480758Z cuda train nanogpt 2024-08-20T23:58:28.4101498Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:58:28.4103308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-20T23:58:28.4104776Z pred = mod(*cloned_inputs) 2024-08-20T23:58:28.4105992Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward 2024-08-20T23:58:28.4107697Z tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) 2024-08-20T23:58:28.4108502Z 2024-08-20T23:58:30.2406085Z W0820 23:58:30.239000 48871 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:59:04.9587888Z pass 2024-08-20T23:59:05.0169210Z TIMING: entire_frame_compile:80.3597 code_gen:21.34044 inductor_compile:41.04632 backend_compile:66.32922 total_wall_time:93.24636 2024-08-20T23:59:05.0171012Z STATS: call_* op count: 858 | FakeTensorMode.__torch_dispatch__:55587 | attempt fast:2561 | fast is_contiguous:2561 | FakeTensor.__torch_dispatch__:12707 | ProxyTorchDispatchMode.__torch_dispatch__:19297 2024-08-20T23:59:05.0173566Z Dynamo produced 3 graphs covering 858 ops with 7 graph breaks (5 unique) 2024-08-20T23:59:12.4603202Z 2024-08-20T23:59:14.9522703Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:14.9523172Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:59:14.9524935Z cuda train nvidia_deeprecommender 2024-08-20T23:59:23.3887764Z W0820 23:59:23.388000 49263 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:59:28.3588685Z pass 2024-08-20T23:59:28.3865453Z TIMING: entire_frame_compile:10.85047 code_gen:4.0018 inductor_compile:5.65275 backend_compile:9.38581 total_wall_time:12.24477 2024-08-20T23:59:28.3867054Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:4792 | attempt fast:210 | fast is_contiguous:210 | ProxyTorchDispatchMode.__torch_dispatch__:1496 | FakeTensor.__torch_dispatch__:1169 2024-08-20T23:59:28.3868344Z Dynamo produced 3 graphs covering 70 ops with 7 graph breaks (5 unique) 2024-08-20T23:59:32.4056411Z 2024-08-20T23:59:33.3670983Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:33.3671475Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:33.3672163Z cuda train opacus_cifar10 2024-08-20T23:59:33.3687266Z Traceback (most recent call last): 2024-08-20T23:59:33.3688247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:59:33.3689103Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:59:33.3689968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:59:33.3690811Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:59:33.3693477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:59:33.3694279Z torch.autograd.backward( 2024-08-20T23:59:33.3695148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:59:33.3695959Z _engine_run_backward( 2024-08-20T23:59:33.3696859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:59:33.3698018Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:59:33.3699500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 98, in __call__ 2024-08-20T23:59:33.3700518Z return self.hook(module, *args, **kwargs) 2024-08-20T23:59:33.3701632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 328, in capture_backprops_hook 2024-08-20T23:59:33.3702659Z activations, backprops = self.rearrange_grad_samples( 2024-08-20T23:59:33.3703803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 384, in rearrange_grad_samples 2024-08-20T23:59:33.3704860Z raise ValueError( 2024-08-20T23:59:33.3705702Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-08-20T23:59:33.3706394Z 2024-08-20T23:59:33.3706687Z The above exception was the direct cause of the following exception: 2024-08-20T23:59:33.3707142Z 2024-08-20T23:59:33.3707299Z Traceback (most recent call last): 2024-08-20T23:59:33.3707960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:59:33.3708618Z ) = runner.load_model( 2024-08-20T23:59:33.3709280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:59:33.3710045Z self.validate_model(model, example_inputs) 2024-08-20T23:59:33.3710797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:59:33.3711559Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:59:33.3712022Z RuntimeError: Eager run failed 2024-08-20T23:59:33.3712283Z 2024-08-20T23:59:33.3712398Z eager_fail_to_run 2024-08-20T23:59:36.9246114Z 2024-08-20T23:59:37.7695990Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:37.7696724Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:37.7697427Z cuda train phlippe_densenet 2024-08-21T00:00:34.3353953Z pass 2024-08-21T00:00:34.3359726Z TIMING: entire_frame_compile:53.17553 code_gen:12.14539 inductor_compile:23.35201 backend_compile:40.81254 total_wall_time:66.43448 2024-08-21T00:00:34.3361956Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:34800 | attempt fast:1414 | fast is_contiguous:1366 | ProxyTorchDispatchMode.__torch_dispatch__:11247 | slow no contiguity match:48 | FakeTensor.__torch_dispatch__:5672 2024-08-21T00:00:34.3363430Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-08-21T00:00:40.1887539Z 2024-08-21T00:00:40.9227635Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:40.9228122Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:40.9231252Z cuda train phlippe_resnet 2024-08-21T00:01:01.1721303Z pass 2024-08-21T00:01:01.1722227Z TIMING: entire_frame_compile:18.15262 code_gen:3.28154 inductor_compile:7.41535 backend_compile:14.09149 total_wall_time:22.45859 2024-08-21T00:01:01.1724001Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:12919 | attempt fast:567 | fast is_contiguous:567 | ProxyTorchDispatchMode.__torch_dispatch__:4228 | FakeTensor.__torch_dispatch__:2127 2024-08-21T00:01:01.1725727Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-08-21T00:01:05.7013972Z 2024-08-21T00:01:07.0012795Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:07.0013691Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:01:07.0014592Z cuda train pytorch_CycleGAN_and_pix2pix 2024-08-21T00:01:43.6539995Z pass 2024-08-21T00:01:43.6629396Z TIMING: entire_frame_compile:11.48256 code_gen:10.5852 inductor_compile:16.43132 backend_compile:9.3185 total_wall_time:23.12781 2024-08-21T00:01:43.6630800Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:13910 | ProxyTorchDispatchMode.__torch_dispatch__:7137 | FakeTensor.__torch_dispatch__:3594 2024-08-21T00:01:43.6631921Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-08-21T00:01:48.3421979Z 2024-08-21T00:01:49.8665359Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:49.8665840Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:01:49.8666288Z cuda train pytorch_stargan 2024-08-21T00:02:44.1587821Z pass 2024-08-21T00:02:44.1664593Z TIMING: entire_frame_compile:26.13446 code_gen:4.16974 inductor_compile:10.76357 backend_compile:21.20755 total_wall_time:31.51532 2024-08-21T00:02:44.1666187Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:20314 | attempt fast:830 | fast is_contiguous:830 | FakeTensor.__torch_dispatch__:2215 | ProxyTorchDispatchMode.__torch_dispatch__:5359 2024-08-21T00:02:44.1667488Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-21T00:02:49.1047544Z 2024-08-21T00:02:50.0500845Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:02:50.0501736Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:02:50.0502600Z cuda train pytorch_unet 2024-08-21T00:03:52.8855328Z W0821 00:03:52.884000 51345 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:04:13.1212389Z pass_due_to_skip 2024-08-21T00:04:13.1316450Z TIMING: entire_frame_compile:50.76175 code_gen:20.84293 inductor_compile:31.15661 backend_compile:43.13886 total_wall_time:63.9035 2024-08-21T00:04:13.1318569Z STATS: call_* op count: 314 | FakeTensorMode.__torch_dispatch__:24690 | attempt fast:856 | fast is_contiguous:848 | ProxyTorchDispatchMode.__torch_dispatch__:9051 | slow no contiguity match:8 | FakeTensor.__torch_dispatch__:6599 2024-08-21T00:04:13.1320178Z Dynamo produced 3 graphs covering 314 ops with 7 graph breaks (5 unique) 2024-08-21T00:04:18.8118823Z 2024-08-21T00:04:21.2379758Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:04:21.2380304Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:04:21.2380751Z cuda train resnet152 2024-08-21T00:06:29.9038432Z W0821 00:06:29.902000 52470 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:08:31.1001330Z pass 2024-08-21T00:08:31.1372790Z TIMING: entire_frame_compile:229.98219 code_gen:60.08177 inductor_compile:110.63228 backend_compile:183.32882 total_wall_time:259.91593 2024-08-21T00:08:31.1374864Z STATS: call_* op count: 1937 | FakeTensorMode.__torch_dispatch__:138184 | attempt fast:4318 | fast is_contiguous:4318 | ProxyTorchDispatchMode.__torch_dispatch__:53198 | FakeTensor.__torch_dispatch__:39757 2024-08-21T00:08:31.1376214Z Dynamo produced 3 graphs covering 1937 ops with 7 graph breaks (5 unique) 2024-08-21T00:08:45.8573036Z 2024-08-21T00:08:47.1264546Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:08:47.1265038Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:08:47.1265473Z cuda train resnet18 2024-08-21T00:09:10.8912162Z pass 2024-08-21T00:09:10.8984555Z TIMING: entire_frame_compile:20.42196 code_gen:5.89623 inductor_compile:10.8868 backend_compile:16.8763 total_wall_time:27.1623 2024-08-21T00:09:10.8986363Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:12423 | attempt fast:598 | fast is_contiguous:598 | ProxyTorchDispatchMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:2149 2024-08-21T00:09:10.8987663Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-08-21T00:09:15.5408367Z 2024-08-21T00:09:17.4007211Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:09:17.4007888Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:09:17.4008561Z cuda train resnet50 2024-08-21T00:10:10.0592969Z pass 2024-08-21T00:10:10.0777503Z TIMING: entire_frame_compile:45.19812 code_gen:9.31086 inductor_compile:20.51815 backend_compile:36.37291 total_wall_time:57.90945 2024-08-21T00:10:10.0779118Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:31341 | attempt fast:1496 | fast is_contiguous:1496 | ProxyTorchDispatchMode.__torch_dispatch__:11011 | FakeTensor.__torch_dispatch__:5583 2024-08-21T00:10:10.0780462Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-08-21T00:10:16.0199237Z 2024-08-21T00:10:18.1699643Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:10:18.1700222Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:10:18.1700700Z cuda train resnet50_quantized_qat 2024-08-21T00:10:18.1701170Z Traceback (most recent call last): 2024-08-21T00:10:18.1701895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:10:18.1702652Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:10:18.1703507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-21T00:10:18.1704450Z pred = mod(*cloned_inputs) 2024-08-21T00:10:18.1706281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-21T00:10:18.1707459Z return self._wrapped_call(self, *args, **kwargs) 2024-08-21T00:10:18.1708679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-21T00:10:18.1709561Z raise e 2024-08-21T00:10:18.1710322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-21T00:10:18.1711286Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-21T00:10:18.1712538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:10:18.1713811Z return self._call_impl(*args, **kwargs) 2024-08-21T00:10:18.1715203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:10:18.1716406Z return forward_call(*args, **kwargs) 2024-08-21T00:10:18.1717058Z File ".3", line 167, in forward 2024-08-21T00:10:18.1717997Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-08-21T00:10:18.1720106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:10:18.1721409Z return self._call_impl(*args, **kwargs) 2024-08-21T00:10:18.1723128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:10:18.1724389Z return forward_call(*args, **kwargs) 2024-08-21T00:10:18.1725863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-21T00:10:18.1727248Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-21T00:10:18.1728090Z RuntimeError: expected scalar type Float but found Half 2024-08-21T00:10:18.1728675Z 2024-08-21T00:10:18.1729103Z The above exception was the direct cause of the following exception: 2024-08-21T00:10:18.1729783Z 2024-08-21T00:10:18.1730005Z Traceback (most recent call last): 2024-08-21T00:10:18.1731002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:10:18.1731980Z ) = runner.load_model( 2024-08-21T00:10:18.1732958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:10:18.1734129Z self.validate_model(model, example_inputs) 2024-08-21T00:10:18.1735300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:10:18.1736485Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:10:18.1737179Z RuntimeError: Eager run failed 2024-08-21T00:10:18.1737590Z 2024-08-21T00:10:18.1737756Z eager_fail_to_run 2024-08-21T00:10:21.8206296Z 2024-08-21T00:10:23.7977850Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:10:23.7978335Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:10:23.7978772Z cuda train resnext50_32x4d 2024-08-21T00:11:16.2439407Z W0821 00:11:16.243000 53247 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:11:57.2551170Z pass 2024-08-21T00:11:57.2671786Z TIMING: entire_frame_compile:82.26313 code_gen:22.79345 inductor_compile:41.6949 backend_compile:65.90456 total_wall_time:94.71946 2024-08-21T00:11:57.2676593Z STATS: call_* op count: 679 | FakeTensorMode.__torch_dispatch__:48000 | attempt fast:1496 | fast is_contiguous:1496 | ProxyTorchDispatchMode.__torch_dispatch__:18280 | FakeTensor.__torch_dispatch__:13739 2024-08-21T00:11:57.2678611Z Dynamo produced 3 graphs covering 679 ops with 7 graph breaks (5 unique) 2024-08-21T00:12:04.7821440Z 2024-08-21T00:12:12.4022948Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:12.4023610Z loading model: 0it [00:07, ?it/s] 2024-08-21T00:12:12.4024388Z cuda train sam 2024-08-21T00:12:12.4033962Z Traceback (most recent call last): 2024-08-21T00:12:12.4035023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:12:12.4036189Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:12:12.4037437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-21T00:12:12.4038694Z self.grad_scaler.scale(loss).backward() 2024-08-21T00:12:12.4040732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-21T00:12:12.4041859Z torch.autograd.backward( 2024-08-21T00:12:12.4043159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-21T00:12:12.4044349Z _engine_run_backward( 2024-08-21T00:12:12.4045721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-21T00:12:12.4047419Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-21T00:12:12.4048826Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-08-21T00:12:12.4050021Z 2024-08-21T00:12:12.4050462Z The above exception was the direct cause of the following exception: 2024-08-21T00:12:12.4051131Z 2024-08-21T00:12:12.4051350Z Traceback (most recent call last): 2024-08-21T00:12:12.4052570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:12:12.4053562Z ) = runner.load_model( 2024-08-21T00:12:12.4054548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:12:12.4055688Z self.validate_model(model, example_inputs) 2024-08-21T00:12:12.4056805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:12:12.4057961Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:12:12.4058655Z RuntimeError: Eager run failed 2024-08-21T00:12:12.4059030Z 2024-08-21T00:12:12.4059214Z eager_fail_to_run 2024-08-21T00:12:16.0503978Z 2024-08-21T00:12:17.3577029Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:17.3577723Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:17.3578373Z cuda train shufflenet_v2_x1_0 2024-08-21T00:13:17.2184410Z pass 2024-08-21T00:13:17.2194088Z TIMING: entire_frame_compile:55.77199 code_gen:15.50049 inductor_compile:28.68751 backend_compile:45.78891 total_wall_time:74.36682 2024-08-21T00:13:17.2195825Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:36655 | attempt fast:1580 | fast is_contiguous:1580 | ProxyTorchDispatchMode.__torch_dispatch__:11975 | FakeTensor.__torch_dispatch__:6649 2024-08-21T00:13:17.2197433Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-08-21T00:13:23.5654171Z 2024-08-21T00:13:24.4321926Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:24.4322568Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:24.4323146Z cuda train soft_actor_critic 2024-08-21T00:13:30.8422309Z W0821 00:13:30.841000 54182 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:13:33.5008129Z pass 2024-08-21T00:13:33.5015135Z TIMING: entire_frame_compile:7.0513 code_gen:2.15747 inductor_compile:3.22555 backend_compile:6.12438 total_wall_time:7.98894 2024-08-21T00:13:33.5016921Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:2916 | attempt fast:69 | fast is_contiguous:69 | ProxyTorchDispatchMode.__torch_dispatch__:894 | FakeTensor.__torch_dispatch__:556 2024-08-21T00:13:33.5018193Z Dynamo produced 4 graphs covering 56 ops with 6 graph breaks (5 unique) 2024-08-21T00:13:37.4940430Z 2024-08-21T00:13:38.8083941Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:38.8084532Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:13:38.8085055Z Traceback (most recent call last): 2024-08-21T00:13:38.8085844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in 2024-08-21T00:13:38.8086645Z torchbench_main() 2024-08-21T00:13:38.8087347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torchbench_main 2024-08-21T00:13:38.8088141Z main(TorchBenchmarkRunner(), original_dir) 2024-08-21T00:13:38.8088840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4087, in main 2024-08-21T00:13:38.8091843Z process_entry(0, runner, original_dir, args) 2024-08-21T00:13:38.8092608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4019, in process_entry 2024-08-21T00:13:38.8096162Z return run(runner, args, original_dir) 2024-08-21T00:13:38.8096844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4661, in run 2024-08-21T00:13:38.8101398Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-08-21T00:13:38.8102103Z AssertionError: nothing in example_inputs had a dim with 32 2024-08-21T00:13:39.5570798Z Run failed with return code: 1 2024-08-21T00:13:39.5571259Z Output: None 2024-08-21T00:13:39.5571564Z Error: None 2024-08-21T00:13:42.3764673Z 2024-08-21T00:13:43.5323600Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:43.5324079Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:13:43.5324518Z cuda train squeezenet1_1 2024-08-21T00:14:08.6825123Z pass 2024-08-21T00:14:08.6846963Z TIMING: entire_frame_compile:22.34698 code_gen:11.63837 inductor_compile:15.59734 backend_compile:20.1901 total_wall_time:32.75946 2024-08-21T00:14:08.6848544Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:9229 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | attempt fast:100 | fast is_contiguous:100 | FakeTensor.__torch_dispatch__:948 2024-08-21T00:14:08.6849824Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-08-21T00:14:13.2512317Z 2024-08-21T00:14:14.4983896Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:14:14.4984242Z 2024-08-21T00:14:14.8516851Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-08-21T00:17:56.4822875Z pass 2024-08-21T00:17:56.4874045Z TIMING: entire_frame_compile:130.17088 code_gen:35.31941 inductor_compile:65.14621 backend_compile:103.8094 total_wall_time:152.84581 2024-08-21T00:17:56.4875654Z STATS: call_* op count: 899 | FakeTensorMode.__torch_dispatch__:69727 | attempt fast:2404 | fast is_contiguous:2404 | FakeTensor.__torch_dispatch__:19882 | ProxyTorchDispatchMode.__torch_dispatch__:24017 2024-08-21T00:17:56.4876975Z Dynamo produced 3 graphs covering 899 ops with 7 graph breaks (5 unique) 2024-08-21T00:18:06.6960591Z 2024-08-21T00:18:09.5177378Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:09.5177850Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:18:09.5178312Z cuda train timm_regnet 2024-08-21T00:19:39.1126433Z pass 2024-08-21T00:19:39.1545111Z TIMING: entire_frame_compile:75.2617 code_gen:13.38929 inductor_compile:34.07561 backend_compile:59.11852 total_wall_time:96.10698 2024-08-21T00:19:39.1547018Z STATS: call_* op count: 323 | FakeTensorMode.__torch_dispatch__:53135 | attempt fast:2163 | fast is_contiguous:2163 | ProxyTorchDispatchMode.__torch_dispatch__:17103 | FakeTensor.__torch_dispatch__:9034 2024-08-21T00:19:39.1548342Z Dynamo produced 2 graphs covering 323 ops with 6 graph breaks (5 unique) 2024-08-21T00:19:46.6711778Z 2024-08-21T00:19:48.6809115Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:48.6810109Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:19:48.6811001Z cuda train timm_resnest 2024-08-21T00:20:34.5169501Z W0821 00:20:34.516000 56732 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:20:57.4958563Z pass 2024-08-21T00:20:57.5026977Z TIMING: entire_frame_compile:60.78506 code_gen:25.35987 inductor_compile:37.26112 backend_compile:51.24859 total_wall_time:77.12037 2024-08-21T00:20:57.5028882Z STATS: call_* op count: 401 | FakeTensorMode.__torch_dispatch__:27631 | attempt fast:797 | fast is_contiguous:781 | ProxyTorchDispatchMode.__torch_dispatch__:9728 | slow no contiguity match:16 | FakeTensor.__torch_dispatch__:7416 2024-08-21T00:20:57.5030342Z Dynamo produced 3 graphs covering 401 ops with 7 graph breaks (5 unique) 2024-08-21T00:21:03.9724051Z 2024-08-21T00:21:05.6378564Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:21:05.6379113Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:21:05.6379674Z cuda train timm_vision_transformer 2024-08-21T00:21:57.0206596Z pass 2024-08-21T00:21:57.0380059Z TIMING: entire_frame_compile:47.09846 code_gen:10.02079 inductor_compile:22.44462 backend_compile:38.74945 total_wall_time:60.33573 2024-08-21T00:21:57.0383465Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:37904 | attempt fast:2319 | fast is_contiguous:2319 | FakeTensor.__torch_dispatch__:5276 | ProxyTorchDispatchMode.__torch_dispatch__:10943 2024-08-21T00:21:57.0384911Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique) 2024-08-21T00:22:03.2810245Z 2024-08-21T00:22:15.9101484Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:15.9102079Z loading model: 0it [00:12, ?it/s] 2024-08-21T00:22:15.9102627Z cuda train timm_vision_transformer_large 2024-08-21T00:22:15.9104527Z pass_due_to_skip 2024-08-21T00:22:16.1783683Z TIMING: 2024-08-21T00:22:16.1784105Z STATS: call_* op count: 0 2024-08-21T00:22:16.1784962Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:22:19.8660478Z 2024-08-21T00:22:21.9071105Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:21.9071641Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:22:21.9072424Z cuda train timm_vovnet 2024-08-21T00:23:17.7059290Z pass 2024-08-21T00:23:17.7243013Z TIMING: entire_frame_compile:46.88544 code_gen:16.03335 inductor_compile:26.85753 backend_compile:39.49624 total_wall_time:64.80787 2024-08-21T00:23:17.7244764Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:25917 | attempt fast:1196 | fast is_contiguous:1136 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | slow no contiguity match:60 | FakeTensor.__torch_dispatch__:4486 2024-08-21T00:23:17.7246231Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique) 2024-08-21T00:23:23.6156725Z 2024-08-21T00:23:25.6569906Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:25.6570456Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:23:25.6570988Z cuda train torch_multimodal_clip 2024-08-21T00:24:34.9416130Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_158) 2024-08-21T00:25:00.8865240Z W0821 00:25:00.885000 58156 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:26:10.5040736Z pass 2024-08-21T00:26:10.5977108Z TIMING: entire_frame_compile:155.02932 code_gen:48.63253 inductor_compile:84.77507 backend_compile:128.01129 total_wall_time:178.66008 2024-08-21T00:26:10.5978865Z STATS: call_* op count: 1404 | FakeTensorMode.__torch_dispatch__:104457 | attempt fast:2217 | fast is_contiguous:2135 | FakeTensor.__torch_dispatch__:24696 | slow no contiguity match:82 | ProxyTorchDispatchMode.__torch_dispatch__:40037 2024-08-21T00:26:10.5980336Z Dynamo produced 3 graphs covering 1404 ops with 7 graph breaks (5 unique) 2024-08-21T00:26:22.2916525Z 2024-08-21T00:26:22.9761575Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:26:22.9762045Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:26:22.9762514Z cuda train tts_angular 2024-08-21T00:26:26.6143767Z W0821 00:26:26.613000 58789 torch/_logging/_internal.py:1058] [10/0] Profiler function will be ignored 2024-08-21T00:26:32.0971775Z pass 2024-08-21T00:26:32.1038122Z TIMING: entire_frame_compile:6.92115 inductor_compile:5.24962 backend_compile:6.0663 code_gen:3.9206 total_wall_time:7.20374 2024-08-21T00:26:32.1039479Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:2216 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:906 2024-08-21T00:26:32.1040579Z Dynamo produced 5 graphs covering 70 ops with 9 graph breaks (6 unique) 2024-08-21T00:26:36.1048298Z 2024-08-21T00:26:38.7728889Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:26:38.7729509Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:26:38.7731252Z cuda train vgg16 2024-08-21T00:27:17.1586039Z pass 2024-08-21T00:27:17.1806940Z TIMING: entire_frame_compile:15.15722 code_gen:7.13072 inductor_compile:10.25972 backend_compile:13.88405 total_wall_time:21.9469 2024-08-21T00:27:17.1810073Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:6735 | attempt fast:174 | fast is_contiguous:174 | ProxyTorchDispatchMode.__torch_dispatch__:1686 | FakeTensor.__torch_dispatch__:596 2024-08-21T00:27:17.1812625Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-08-21T00:27:21.4656341Z 2024-08-21T00:27:24.8268006Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:27:24.8268498Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:27:24.8268945Z cuda train vision_maskrcnn 2024-08-21T00:27:59.3619341Z W0821 00:27:59.361000 59352 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:27:59.3623295Z W0821 00:27:59.361000 59352 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:27:59.3630043Z W0821 00:27:59.362000 59352 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:27:59.3636666Z W0821 00:27:59.363000 59352 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:27:59.3643488Z W0821 00:27:59.363000 59352 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:28:01.4017451Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 2024-08-21T00:28:01.4019166Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward 2024-08-21T00:28:01.4020471Z anchors = self.anchor_generator(images, features) 2024-08-21T00:28:01.4021897Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward 2024-08-21T00:28:01.4023129Z self.set_cell_anchors(dtype, device) 2024-08-21T00:28:01.4024713Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors 2024-08-21T00:28:01.4026357Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-21T00:28:01.4027723Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in 2024-08-21T00:28:01.4028942Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-21T00:28:01.4029523Z 2024-08-21T00:28:03.1572825Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-08-21T00:28:03.1574328Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-21T00:28:03.1575697Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] or: 2024-08-21T00:28:03.1576925Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-21T00:28:03.1578314Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] to include these operations in the captured graph. 2024-08-21T00:28:03.1579376Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:28:03.1580404Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break: from user code at: 2024-08-21T00:28:03.1582694Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-21T00:28:03.1585010Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-21T00:28:03.1587159Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-21T00:28:03.1588994Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-21T00:28:03.1590900Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-21T00:28:03.1592525Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] self.scales, self.map_levels = _setup_scales( 2024-08-21T00:28:03.1594382Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-21T00:28:03.1596183Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-21T00:28:03.1598402Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-21T00:28:03.1600362Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-21T00:28:03.1602329Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-21T00:28:03.1604053Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-21T00:28:03.1605136Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:28:03.1605947Z W0821 00:28:03.156000 59352 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:28:17.8427321Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:28:17.8428730Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:28:17.8429858Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:28:17.8430454Z 2024-08-21T00:28:23.7102757Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:28:23.7105317Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:28:23.7106924Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:28:23.7107517Z 2024-08-21T00:28:29.5309686Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:28:29.5311005Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:28:29.5312153Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:28:29.5312738Z 2024-08-21T00:29:07.1130953Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:07.1133266Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:29:07.1135301Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:29:07.1136497Z 2024-08-21T00:29:16.6895201Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:16.6897704Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:29:16.6899922Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:29:16.6901100Z 2024-08-21T00:29:30.2357792Z W0821 00:29:30.234000 59352 torch/fx/experimental/symbolic_shapes.py:4543] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s3, s4), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-08-21T00:29:32.4153641Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:32.4155138Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:29:32.4156283Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:29:32.4156909Z 2024-08-21T00:29:36.9515731Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:36.9517385Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:29:36.9518569Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:29:36.9519158Z 2024-08-21T00:29:43.5087656Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:43.5090495Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-08-21T00:29:43.5092836Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-08-21T00:29:43.5093630Z 2024-08-21T00:29:44.4865343Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:44.4867026Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-08-21T00:29:44.4868301Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-08-21T00:29:44.4868787Z 2024-08-21T00:29:45.6036630Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:45.6038565Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804 2024-08-21T00:29:45.6039772Z masks_probs = maskrcnn_inference(mask_logits, labels) 2024-08-21T00:29:45.6041004Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference 2024-08-21T00:29:45.6042120Z mask_prob = mask_prob[index, labels][:, None] 2024-08-21T00:29:45.6042489Z 2024-08-21T00:29:48.8044913Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:48.8046242Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:29:48.8047360Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:29:48.8047943Z 2024-08-21T00:29:52.5533046Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:29:52.5534647Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward 2024-08-21T00:29:52.5535970Z features = self.backbone(images.tensors) 2024-08-21T00:29:52.5537374Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward 2024-08-21T00:29:52.5538361Z x = self.fpn(x) 2024-08-21T00:29:52.5539277Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward 2024-08-21T00:29:52.5540341Z inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest") 2024-08-21T00:29:52.5540804Z 2024-08-21T00:29:57.7187310Z W0821 00:29:57.718000 59352 torch/_dynamo/convert_frame.py:831] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-08-21T00:29:57.7190027Z W0821 00:29:57.718000 59352 torch/_dynamo/convert_frame.py:831] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-08-21T00:29:57.7192914Z W0821 00:29:57.718000 59352 torch/_dynamo/convert_frame.py:831] [30/8] last reason: 30/0: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623 2024-08-21T00:29:57.7195059Z W0821 00:29:57.718000 59352 torch/_dynamo/convert_frame.py:831] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-08-21T00:29:57.7197429Z W0821 00:29:57.718000 59352 torch/_dynamo/convert_frame.py:831] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-08-21T00:30:05.6622797Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:05.6624676Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:05.6626152Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:05.6627549Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:05.6628778Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:05.6629703Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:05.6630505Z return input[ 2024-08-21T00:30:05.6630863Z 2024-08-21T00:30:11.0633826Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:11.0635649Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:11.0637248Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:11.0639136Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:11.0640483Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:11.0641941Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:11.0643237Z return input[ 2024-08-21T00:30:11.0643559Z 2024-08-21T00:30:17.6845232Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:17.6846970Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:17.6848306Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:17.6849921Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:17.6850962Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:17.6851896Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:17.6852701Z return input[ 2024-08-21T00:30:17.6852904Z 2024-08-21T00:30:30.1836182Z W0821 00:30:30.182000 59352 torch/fx/experimental/symbolic_shapes.py:4543] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-08-21T00:30:31.7095995Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:31.7097772Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:31.7098929Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:31.7100194Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:31.7101075Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:31.7101994Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:31.7102803Z return input[ 2024-08-21T00:30:31.7103001Z 2024-08-21T00:30:42.3963350Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:42.3965042Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:42.3966130Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:42.3967421Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:42.3968302Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:42.3969221Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:42.3970020Z return input[ 2024-08-21T00:30:42.3970213Z 2024-08-21T00:30:51.2700937Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:51.2702601Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:51.2704122Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:51.2705597Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:51.2706481Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:51.2707587Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:51.2708403Z return input[ 2024-08-21T00:30:51.2708596Z 2024-08-21T00:30:55.4927746Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:55.4929445Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:55.4930892Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:55.4932184Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:55.4933107Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:55.4934024Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:55.4934835Z return input[ 2024-08-21T00:30:55.4935031Z 2024-08-21T00:30:56.6103980Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:56.6105570Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:30:56.6106678Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:30:56.6107929Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:30:56.6108812Z v1 = masked_index(y_low, x_low) 2024-08-21T00:30:56.6109728Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:30:56.6110552Z return input[ 2024-08-21T00:30:56.6110748Z 2024-08-21T00:30:56.9527627Z pass 2024-08-21T00:30:57.0150040Z TIMING: entire_frame_compile:192.54011 inductor_compile:92.90608 backend_compile:145.72343 code_gen:55.74471 total_wall_time:227.04322 2024-08-21T00:30:57.0152203Z STATS: call_* op count: 2436 | FakeTensorMode.__torch_dispatch__:137862 | FakeTensor.__torch_dispatch__:12951 | ProxyTorchDispatchMode.__torch_dispatch__:45360 | attempt fast:2589 | slow no contiguity match:274 | fast is_contiguous:1813 | slow both tensors nontrivially broadcast:502 2024-08-21T00:30:57.0153905Z Dynamo produced 51 graphs covering 2436 ops with 33 graph breaks (8 unique) 2024-08-21T00:31:07.6044143Z 2024-08-21T00:31:10.5170358Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:31:10.5171488Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:31:10.5173386Z cuda train yolov3 2024-08-21T00:32:26.3422913Z W0821 00:32:26.341000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:26.3425926Z W0821 00:32:26.342000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:26.6855523Z W0821 00:32:26.684000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:26.6857440Z W0821 00:32:26.685000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:27.0318429Z W0821 00:32:27.031000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:27.0320377Z W0821 00:32:27.031000 60826 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:32:45.3282193Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_359). Found from : 2024-08-21T00:32:45.3283657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:32:45.3285582Z pred = mod(*cloned_inputs) 2024-08-21T00:32:45.3287036Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-21T00:32:45.3287944Z return self.forward_once(x) 2024-08-21T00:32:45.3288983Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-08-21T00:32:45.3289970Z yolo_out.append(module(x, out)) 2024-08-21T00:32:45.3290919Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-08-21T00:32:45.3291878Z self.create_grids((nx, ny), p.device) 2024-08-21T00:32:45.3292859Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-08-21T00:32:45.3293876Z self.anchor_vec = self.anchor_vec.to(device) 2024-08-21T00:32:45.3294233Z 2024-08-21T00:33:28.5791255Z W0821 00:33:28.578000 60826 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:36:12.0858009Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:36:12.0859186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:36:12.0861897Z pred = mod(*cloned_inputs) 2024-08-21T00:36:12.0862873Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-21T00:36:12.0863749Z return self.forward_once(x) 2024-08-21T00:36:12.0864626Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 292, in forward_once 2024-08-21T00:36:12.0865615Z x = module(x) 2024-08-21T00:36:12.0865816Z 2024-08-21T00:36:15.0705862Z pass 2024-08-21T00:36:15.1095177Z TIMING: entire_frame_compile:283.17776 code_gen:98.65607 inductor_compile:158.58877 backend_compile:237.74123 total_wall_time:360.74104 2024-08-21T00:36:15.1096956Z STATS: call_* op count: 1377 | FakeTensorMode.__torch_dispatch__:131273 | attempt fast:6988 | fast is_contiguous:6976 | FakeTensor.__torch_dispatch__:34810 | ProxyTorchDispatchMode.__torch_dispatch__:44439 | slow no contiguity match:12 2024-08-21T00:36:15.1098426Z Dynamo produced 4 graphs covering 1377 ops with 8 graph breaks (5 unique) 2024-08-21T00:36:29.1467245Z accuracy pass_rate=78.38% 2024-08-21T00:36:29.1469678Z calls_captured gmean=0.00x mean=422.054x 2024-08-21T00:36:29.1474084Z unique_graphs gmean=0.00x mean=3.649x 2024-08-21T00:36:29.1477549Z graph_breaks gmean=0.00x mean=6.135x 2024-08-21T00:36:29.1481605Z unique_graph_breaks gmean=0.00x mean=4.243x 2024-08-21T00:36:29.1484892Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:36:29.1488290Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:36:29.1491895Z cudagraph_skips gmean=0.00x mean=0.811x 2024-08-21T00:36:30.7984942Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv 2024-08-21T00:36:31.1326035Z llava XFAIL 2024-08-21T00:36:31.1329706Z maml_omniglot PASS 2024-08-21T00:36:31.1334194Z mnasnet1_0 PASS 2024-08-21T00:36:31.1338590Z mobilenet_v2 PASS 2024-08-21T00:36:31.1342985Z mobilenet_v2_quantized_qat XFAIL 2024-08-21T00:36:31.1347732Z mobilenet_v3_large PASS 2024-08-21T00:36:31.1352186Z moco PASS 2024-08-21T00:36:31.1356613Z nanogpt PASS 2024-08-21T00:36:31.1360861Z nvidia_deeprecommender PASS 2024-08-21T00:36:31.1365059Z opacus_cifar10 XFAIL 2024-08-21T00:36:31.1369421Z phlippe_densenet PASS 2024-08-21T00:36:31.1374420Z phlippe_resnet PASS 2024-08-21T00:36:31.1379082Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:36:31.1383262Z pytorch_stargan PASS 2024-08-21T00:36:31.1388029Z pytorch_unet XFAIL 2024-08-21T00:36:31.1392360Z resnet152 PASS 2024-08-21T00:36:31.1396782Z resnet18 PASS 2024-08-21T00:36:31.1401299Z resnet50 PASS 2024-08-21T00:36:31.1405325Z resnet50_quantized_qat XFAIL 2024-08-21T00:36:31.1409716Z resnext50_32x4d PASS 2024-08-21T00:36:31.1414314Z sam XFAIL 2024-08-21T00:36:31.1418495Z shufflenet_v2_x1_0 PASS 2024-08-21T00:36:31.1422835Z soft_actor_critic PASS 2024-08-21T00:36:31.1427840Z squeezenet1_1 PASS 2024-08-21T00:36:31.1431807Z stable_diffusion_text_encoder PASS 2024-08-21T00:36:31.1436044Z stable_diffusion_unet XFAIL 2024-08-21T00:36:31.1440490Z timm_efficientnet PASS 2024-08-21T00:36:31.1444956Z timm_regnet PASS 2024-08-21T00:36:31.1449215Z timm_resnest PASS 2024-08-21T00:36:31.1453759Z timm_vision_transformer PASS 2024-08-21T00:36:31.1458003Z timm_vision_transformer_large XFAIL 2024-08-21T00:36:31.1462512Z timm_vovnet PASS 2024-08-21T00:36:31.1466988Z torch_multimodal_clip PASS 2024-08-21T00:36:31.1471427Z tts_angular PASS 2024-08-21T00:36:31.1476068Z vgg16 PASS 2024-08-21T00:36:31.1480510Z vision_maskrcnn PASS 2024-08-21T00:36:31.1484702Z yolov3 PASS 2024-08-21T00:36:31.2002053Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv 2024-08-21T00:36:31.5365878Z llava PASS 2024-08-21T00:36:31.5369470Z maml_omniglot PASS 2024-08-21T00:36:31.5374854Z mnasnet1_0 PASS 2024-08-21T00:36:31.5380406Z mobilenet_v2 PASS 2024-08-21T00:36:31.5384662Z mobilenet_v2_quantized_qat PASS 2024-08-21T00:36:31.5389739Z mobilenet_v3_large PASS 2024-08-21T00:36:31.5394515Z moco PASS 2024-08-21T00:36:31.5399565Z nanogpt PASS 2024-08-21T00:36:31.5404289Z nvidia_deeprecommender PASS 2024-08-21T00:36:31.5408983Z opacus_cifar10 PASS 2024-08-21T00:36:31.5413886Z phlippe_densenet PASS 2024-08-21T00:36:31.5418551Z phlippe_resnet PASS 2024-08-21T00:36:31.5423434Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:36:31.5428524Z pytorch_stargan PASS 2024-08-21T00:36:31.5433269Z pytorch_unet PASS 2024-08-21T00:36:31.5438042Z resnet152 PASS 2024-08-21T00:36:31.5442647Z resnet18 PASS 2024-08-21T00:36:31.5447418Z resnet50 PASS 2024-08-21T00:36:31.5452384Z resnet50_quantized_qat PASS 2024-08-21T00:36:31.5456964Z resnext50_32x4d PASS 2024-08-21T00:36:31.5461694Z sam PASS 2024-08-21T00:36:31.5466655Z shufflenet_v2_x1_0 PASS 2024-08-21T00:36:31.5471466Z soft_actor_critic PASS 2024-08-21T00:36:31.5476732Z squeezenet1_1 PASS 2024-08-21T00:36:31.5481644Z stable_diffusion_text_encoder PASS 2024-08-21T00:36:31.5486446Z stable_diffusion_unet PASS 2024-08-21T00:36:31.5491336Z timm_efficientnet PASS 2024-08-21T00:36:31.5496213Z timm_regnet PASS 2024-08-21T00:36:31.5501161Z timm_resnest PASS 2024-08-21T00:36:31.5506657Z timm_vision_transformer PASS 2024-08-21T00:36:31.5511539Z timm_vision_transformer_large PASS 2024-08-21T00:36:31.5516240Z timm_vovnet PASS 2024-08-21T00:36:31.5521064Z torch_multimodal_clip PASS 2024-08-21T00:36:31.5525824Z tts_angular PASS 2024-08-21T00:36:31.5530630Z vgg16 PASS 2024-08-21T00:36:31.5535608Z vision_maskrcnn PASS 2024-08-21T00:36:31.5540300Z yolov3 PASS 2024-08-21T00:36:31.6059186Z + cleanup_workspace 2024-08-21T00:36:31.6060423Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:36:31.6061825Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:36:31.6063009Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:36:31.6063666Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:36:31.6065441Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:36:31.6066386Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:36:31.6067106Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:36:32.3751292Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:36:32.3751825Z cat test/**/*_toprint.log || true 2024-08-21T00:36:32.3765599Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.3766099Z env: 2024-08-21T00:36:32.3766369Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.3766820Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.3767560Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.3768201Z ##[endgroup] 2024-08-21T00:36:32.3865141Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:36:32.3899125Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:36:32.3899599Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:36:32.3909104Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.3909604Z env: 2024-08-21T00:36:32.3909873Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.3910325Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.3911055Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.3911712Z MONITOR_SCRIPT_PID: 52519 2024-08-21T00:36:32.3912054Z ##[endgroup] 2024-08-21T00:36:32.4142749Z Prepare all required actions 2024-08-21T00:36:32.4143209Z Getting action download info 2024-08-21T00:36:32.6010499Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:36:32.7567921Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:36:32.7568382Z with: 2024-08-21T00:36:32.7568998Z file-suffix: test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670 2024-08-21T00:36:32.7569737Z s3-bucket: gha-artifacts 2024-08-21T00:36:32.7570087Z env: 2024-08-21T00:36:32.7570351Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.7570800Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.7571529Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.7572440Z ##[endgroup] 2024-08-21T00:36:32.7598695Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:36:32.7599339Z # Remove any previous test jsons if they exist 2024-08-21T00:36:32.7599839Z rm -f test-jsons-*.zip 2024-08-21T00:36:32.7600355Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:36:32.7609976Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.7610470Z env: 2024-08-21T00:36:32.7610739Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.7611189Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.7611920Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.7613075Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670 2024-08-21T00:36:32.7613790Z ##[endgroup] 2024-08-21T00:36:32.8004251Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:36:32.8044602Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:36:32.8050111Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:36:32.8063690Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:36:32.8074991Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:36:32.8107563Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:36:32.8108214Z # Remove any previous test reports if they exist 2024-08-21T00:36:32.8108745Z rm -f test-reports-*.zip 2024-08-21T00:36:32.8109332Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:36:32.8118338Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.8118835Z env: 2024-08-21T00:36:32.8119109Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.8119559Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.8120287Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.8121281Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670 2024-08-21T00:36:32.8121999Z ##[endgroup] 2024-08-21T00:36:32.8386833Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-08-21T00:36:32.8389316Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-21T00:36:32.8390420Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:36:32.8391419Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-08-21T00:36:32.8398370Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-21T00:36:32.8399912Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:36:32.8432434Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:36:32.8433066Z # Remove any previous usage logs if they exist 2024-08-21T00:36:32.8433560Z rm -f logs-*.zip 2024-08-21T00:36:32.8434420Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:36:32.8435205Z # so check to see if the file exists first 2024-08-21T00:36:32.8435697Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:36:32.8436222Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:36:32.8436705Z fi 2024-08-21T00:36:32.8437064Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:36:32.8437625Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:36:32.8438114Z fi 2024-08-21T00:36:32.8446677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.8447166Z env: 2024-08-21T00:36:32.8447430Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.8447872Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.8448603Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.8449580Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670 2024-08-21T00:36:32.8450294Z ##[endgroup] 2024-08-21T00:36:32.8567668Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:36:32.8658953Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:36:32.8659691Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:36:32.8660233Z rm -f debug-*.zip 2024-08-21T00:36:32.8660608Z if [ -d 'test/debug' ]; then 2024-08-21T00:36:32.8661101Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:36:32.8661774Z fi 2024-08-21T00:36:32.8671151Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:32.8671777Z env: 2024-08-21T00:36:32.8672372Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.8672826Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.8673563Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.8674552Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670 2024-08-21T00:36:32.8675259Z ##[endgroup] 2024-08-21T00:36:32.8770641Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:36:32.8771075Z with: 2024-08-21T00:36:32.8771343Z s3-bucket: gha-artifacts 2024-08-21T00:36:32.8771769Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:32.8772570Z retention-days: 14 2024-08-21T00:36:32.8772913Z if-no-files-found: warn 2024-08-21T00:36:32.8773257Z path: test-jsons-*.zip 2024-08-21T00:36:32.8773603Z name: artifact 2024-08-21T00:36:32.8773901Z region: us-east-1 2024-08-21T00:36:32.8774200Z env: 2024-08-21T00:36:32.8774460Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:32.8774898Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:32.8775627Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:32.8776259Z ##[endgroup] 2024-08-21T00:36:33.2616503Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:36:33.2617768Z With the provided path, there will be 1 file uploaded 2024-08-21T00:36:33.2618895Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:33.3805503Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:33.4768247Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:33.5055088Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:36:33.5055542Z with: 2024-08-21T00:36:33.5055829Z s3-bucket: gha-artifacts 2024-08-21T00:36:33.5056268Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:33.5056736Z retention-days: 14 2024-08-21T00:36:33.5057076Z if-no-files-found: error 2024-08-21T00:36:33.5057443Z path: test-reports-*.zip 2024-08-21T00:36:33.5057794Z name: artifact 2024-08-21T00:36:33.5058097Z region: us-east-1 2024-08-21T00:36:33.5058616Z env: 2024-08-21T00:36:33.5058895Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:33.5059349Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:33.5060078Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:33.5060720Z ##[endgroup] 2024-08-21T00:36:33.8612096Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:36:33.8612752Z With the provided path, there will be 1 file uploaded 2024-08-21T00:36:33.8613445Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:33.8669678Z Starting upload of test-reports-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:33.9969668Z Finished upload of test-reports-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:34.0258183Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:36:34.0258625Z with: 2024-08-21T00:36:34.0258933Z s3-bucket: gha-artifacts 2024-08-21T00:36:34.0259373Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:34.0259895Z retention-days: 14 2024-08-21T00:36:34.0260221Z if-no-files-found: ignore 2024-08-21T00:36:34.0260578Z path: logs-*.zip 2024-08-21T00:36:34.0260887Z name: artifact 2024-08-21T00:36:34.0261184Z region: us-east-1 2024-08-21T00:36:34.0261479Z env: 2024-08-21T00:36:34.0261750Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:34.0262200Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:34.0262929Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:34.0263759Z ##[endgroup] 2024-08-21T00:36:34.3823297Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:36:34.3824189Z With the provided path, there will be 1 file uploaded 2024-08-21T00:36:34.3825153Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:34.3882227Z Starting upload of logs-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:34.5550015Z Finished upload of logs-test-dynamic_inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529670.zip 2024-08-21T00:36:34.5838352Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:36:34.5838791Z with: 2024-08-21T00:36:34.5839074Z s3-bucket: gha-artifacts 2024-08-21T00:36:34.5839506Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:36:34.5839970Z retention-days: 14 2024-08-21T00:36:34.5840303Z if-no-files-found: ignore 2024-08-21T00:36:34.5840672Z path: debug-*.zip 2024-08-21T00:36:34.5840980Z name: artifact 2024-08-21T00:36:34.5841272Z region: us-east-1 2024-08-21T00:36:34.5841569Z env: 2024-08-21T00:36:34.5841840Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:34.5842282Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:34.5843006Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:34.5843641Z ##[endgroup] 2024-08-21T00:36:34.9320754Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:36:34.9613775Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:36:34.9614261Z # shellcheck disable=SC2156 2024-08-21T00:36:34.9615043Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:36:34.9625422Z shell: /usr/bin/bash -e {0} 2024-08-21T00:36:34.9625771Z env: 2024-08-21T00:36:34.9626043Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:34.9626510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:34.9627244Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:34.9627893Z ##[endgroup] 2024-08-21T00:36:35.2299155Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:36:35.2299757Z with: 2024-08-21T00:36:35.2300011Z env: 2024-08-21T00:36:35.2300289Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:35.2300737Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:35.2301469Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:35.2302115Z ##[endgroup] 2024-08-21T00:36:35.2321456Z ##[group]Run set -eou pipefail 2024-08-21T00:36:35.2321861Z set -eou pipefail 2024-08-21T00:36:35.2322210Z  2024-08-21T00:36:35.2322722Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:36:35.2323354Z for _ in $(seq 1440); do 2024-08-21T00:36:35.2323851Z  # Break if no ssh session exists anymore 2024-08-21T00:36:35.2324347Z  if [ "$(who)" = "" ]; then 2024-08-21T00:36:35.2324759Z  break 2024-08-21T00:36:35.2325094Z  fi 2024-08-21T00:36:35.2325393Z  echo "." 2024-08-21T00:36:35.2325723Z  sleep 5 2024-08-21T00:36:35.2326041Z done 2024-08-21T00:36:35.2334728Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:35.2335215Z env: 2024-08-21T00:36:35.2335489Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:35.2335940Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:35.2336678Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:35.2337309Z ##[endgroup] 2024-08-21T00:36:35.2367327Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:36:35.2459783Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:36:35.2460708Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:36:35.2461293Z # shellcheck disable=SC2046 2024-08-21T00:36:35.2461766Z docker stop $(docker ps -q) || true 2024-08-21T00:36:35.2462241Z # Prune all of the docker images 2024-08-21T00:36:35.2462677Z docker system prune -af 2024-08-21T00:36:35.2471863Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:35.2472695Z env: 2024-08-21T00:36:35.2472973Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:36:35.2473415Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:36:35.2474138Z DOCKER_CONTAINER_ID: 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:35.2474769Z ##[endgroup] 2024-08-21T00:36:35.9822836Z 1b6b87892a85 2024-08-21T00:36:39.9970991Z Deleted Containers: 2024-08-21T00:36:39.9971613Z 1b6b87892a855f7bdc03cb0c6d3d9bc1de4555e5a719f7ca00504fb918e7cdb8 2024-08-21T00:36:39.9972428Z 2024-08-21T00:36:49.0120234Z Deleted Images: 2024-08-21T00:36:49.0122215Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:36:49.0125100Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:36:49.0126646Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:36:49.0127505Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:36:49.0128361Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:36:49.0129190Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:36:49.0129993Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:36:49.0130821Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:36:49.0131647Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:36:49.0142797Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:36:49.0143642Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:36:49.0144606Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:36:49.0145441Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:36:49.0146251Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:36:49.0147117Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:36:49.0147937Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:36:49.0148759Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:36:49.0149583Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:36:49.0150418Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:36:49.0151235Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:36:49.0152046Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:36:49.0152863Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:36:49.0153689Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:36:49.0154520Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:36:49.0155337Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:36:49.0156155Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:36:49.0157182Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:36:49.0158012Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:36:49.0158837Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:36:49.0159668Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:36:49.0160496Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:36:49.0161316Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:36:49.0162134Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:36:49.0162945Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:36:49.0163757Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:36:49.0164585Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:36:49.0165559Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:36:49.0166430Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:36:49.0167248Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:36:49.0168068Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:36:49.0168888Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:36:49.0169703Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:36:49.0170526Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:36:49.0171359Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:36:49.0172562Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:36:49.0173361Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:36:49.0174358Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:36:49.0175195Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:36:49.0176016Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:36:49.0176815Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:36:49.0177628Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:36:49.0178461Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:36:49.0179283Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:36:49.0180079Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:36:49.0180939Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:36:49.0181774Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:36:49.0182604Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:36:49.0183425Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:36:49.0184307Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:36:49.0185124Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:36:49.0185930Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:36:49.0186738Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:36:49.0187558Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:36:49.0188523Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:36:49.0189354Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:36:49.0190164Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:36:49.0190969Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:36:49.0191789Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:36:49.0192604Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:36:49.0193425Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:36:49.0194252Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:36:49.0195062Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:36:49.0195867Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:36:49.0196894Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:36:49.0198002Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:36:49.0198950Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:36:49.0199784Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:36:49.0200619Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:36:49.0201454Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:36:49.0202274Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:36:49.0203090Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:36:49.0203588Z 2024-08-21T00:36:49.0203737Z Total reclaimed space: 49.95GB 2024-08-21T00:36:49.0270238Z Post job cleanup. 2024-08-21T00:36:49.0320972Z Post job cleanup. 2024-08-21T00:36:49.1218809Z [command]/usr/bin/git version 2024-08-21T00:36:49.1286739Z git version 2.40.1 2024-08-21T00:36:49.1324911Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c1ff70ff-ff23-4062-ae2f-8c72e134817a' before making global git config changes 2024-08-21T00:36:49.1326153Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:36:49.1330174Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:36:49.1378809Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:36:49.1425160Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:36:49.1818299Z Entering 'android/libs/fbjni' 2024-08-21T00:36:49.1884946Z Entering 'third_party/FP16' 2024-08-21T00:36:49.1951664Z Entering 'third_party/FXdiv' 2024-08-21T00:36:49.2019429Z Entering 'third_party/NNPACK' 2024-08-21T00:36:49.2088302Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:36:49.2155800Z Entering 'third_party/XNNPACK' 2024-08-21T00:36:49.2239522Z Entering 'third_party/benchmark' 2024-08-21T00:36:49.2307800Z Entering 'third_party/cpp-httplib' 2024-08-21T00:36:49.2374379Z Entering 'third_party/cpuinfo' 2024-08-21T00:36:49.2441603Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:36:49.2509447Z Entering 'third_party/cutlass' 2024-08-21T00:36:49.2584311Z Entering 'third_party/eigen' 2024-08-21T00:36:49.2652850Z Entering 'third_party/fbgemm' 2024-08-21T00:36:49.2720414Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:36:49.2787242Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:36:49.2853633Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:36:49.2927463Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:36:49.2995403Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:36:49.3062949Z Entering 'third_party/flatbuffers' 2024-08-21T00:36:49.3133621Z Entering 'third_party/fmt' 2024-08-21T00:36:49.3206195Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:36:49.3275233Z Entering 'third_party/gloo' 2024-08-21T00:36:49.3342557Z Entering 'third_party/googletest' 2024-08-21T00:36:49.3411698Z Entering 'third_party/ideep' 2024-08-21T00:36:49.3477512Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:36:49.3552087Z Entering 'third_party/ittapi' 2024-08-21T00:36:49.3618516Z Entering 'third_party/kineto' 2024-08-21T00:36:49.3687204Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:36:49.3751916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:36:49.3820293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:36:49.3887206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:36:49.3953469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:36:49.4018622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:36:49.4090467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:36:49.4157336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:36:49.4225179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:36:49.4293951Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:36:49.4369005Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:36:49.4433908Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:36:49.4503456Z Entering 'third_party/mimalloc' 2024-08-21T00:36:49.4570820Z Entering 'third_party/nccl/nccl' 2024-08-21T00:36:49.4641011Z Entering 'third_party/nlohmann' 2024-08-21T00:36:49.4709178Z Entering 'third_party/onnx' 2024-08-21T00:36:49.4789239Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:36:49.4858541Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:36:49.4930151Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:36:49.4998277Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:36:49.5064952Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:36:49.5132080Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:36:49.5199811Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:36:49.5267193Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:36:49.5332449Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:36:49.5398894Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:36:49.5460861Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:36:49.5527720Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:36:49.5596190Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:36:49.5690528Z Entering 'third_party/pocketfft' 2024-08-21T00:36:49.5756224Z Entering 'third_party/protobuf' 2024-08-21T00:36:49.5823929Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:36:49.5888312Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:36:49.5955697Z Entering 'third_party/psimd' 2024-08-21T00:36:49.6020843Z Entering 'third_party/pthreadpool' 2024-08-21T00:36:49.6085425Z Entering 'third_party/pybind11' 2024-08-21T00:36:49.6150372Z Entering 'third_party/python-peachpy' 2024-08-21T00:36:49.6216989Z Entering 'third_party/sleef' 2024-08-21T00:36:49.6282399Z Entering 'third_party/tensorpipe' 2024-08-21T00:36:49.6346101Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:36:49.6412420Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:36:49.6475991Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:36:49.6545032Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:36:49.6606849Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:36:49.6699698Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:36:49.6735904Z http.https://github.com/.extraheader 2024-08-21T00:36:49.6747320Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:36:49.6795865Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:36:49.7169397Z Entering 'android/libs/fbjni' 2024-08-21T00:36:49.7212502Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7257298Z Entering 'third_party/FP16' 2024-08-21T00:36:49.7301899Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7345902Z Entering 'third_party/FXdiv' 2024-08-21T00:36:49.7390238Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7433016Z Entering 'third_party/NNPACK' 2024-08-21T00:36:49.7480584Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7523478Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:36:49.7566696Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7609655Z Entering 'third_party/XNNPACK' 2024-08-21T00:36:49.7652144Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7714148Z Entering 'third_party/benchmark' 2024-08-21T00:36:49.7761883Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7805418Z Entering 'third_party/cpp-httplib' 2024-08-21T00:36:49.7847159Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7889690Z Entering 'third_party/cpuinfo' 2024-08-21T00:36:49.7932738Z http.https://github.com/.extraheader 2024-08-21T00:36:49.7977479Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:36:49.8023420Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8067306Z Entering 'third_party/cutlass' 2024-08-21T00:36:49.8110890Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8160668Z Entering 'third_party/eigen' 2024-08-21T00:36:49.8204601Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8249378Z Entering 'third_party/fbgemm' 2024-08-21T00:36:49.8293060Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8336036Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:36:49.8378392Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8421449Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:36:49.8464267Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8510387Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:36:49.8551676Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8600766Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:36:49.8642238Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8685003Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:36:49.8727153Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8773109Z Entering 'third_party/flatbuffers' 2024-08-21T00:36:49.8815932Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8861184Z Entering 'third_party/fmt' 2024-08-21T00:36:49.8905408Z http.https://github.com/.extraheader 2024-08-21T00:36:49.8948166Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:36:49.8994338Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9039176Z Entering 'third_party/gloo' 2024-08-21T00:36:49.9083803Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9127028Z Entering 'third_party/googletest' 2024-08-21T00:36:49.9170007Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9213463Z Entering 'third_party/ideep' 2024-08-21T00:36:49.9257040Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9298907Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:36:49.9341421Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9394901Z Entering 'third_party/ittapi' 2024-08-21T00:36:49.9438019Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9481723Z Entering 'third_party/kineto' 2024-08-21T00:36:49.9524935Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9568604Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:36:49.9611842Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9654285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:36:49.9697860Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9742925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:36:49.9787448Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9832163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:36:49.9875969Z http.https://github.com/.extraheader 2024-08-21T00:36:49.9920541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:36:49.9965071Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0009293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:36:50.0054288Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0102331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:36:50.0145953Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0190071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:36:50.0232819Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0279211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:36:50.0321600Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0365466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:36:50.0413446Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0460870Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:36:50.0504725Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0547929Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:36:50.0591043Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0637361Z Entering 'third_party/mimalloc' 2024-08-21T00:36:50.0682360Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0726691Z Entering 'third_party/nccl/nccl' 2024-08-21T00:36:50.0769631Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0813342Z Entering 'third_party/nlohmann' 2024-08-21T00:36:50.0863502Z http.https://github.com/.extraheader 2024-08-21T00:36:50.0913402Z Entering 'third_party/onnx' 2024-08-21T00:36:50.0961684Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1019350Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:36:50.1062615Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1106507Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:36:50.1149533Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1197676Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:36:50.1241453Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1287725Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:36:50.1329514Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1372585Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:36:50.1414959Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1458040Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:36:50.1500560Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1542952Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:36:50.1585233Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1628948Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:36:50.1670987Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1714237Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:36:50.1756476Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1799574Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:36:50.1841420Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1882801Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:36:50.1925535Z http.https://github.com/.extraheader 2024-08-21T00:36:50.1971236Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:36:50.2013272Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2059279Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:36:50.2101840Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2166876Z Entering 'third_party/pocketfft' 2024-08-21T00:36:50.2210834Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2253154Z Entering 'third_party/protobuf' 2024-08-21T00:36:50.2300976Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2345956Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:36:50.2388668Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2431281Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:36:50.2473984Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2519997Z Entering 'third_party/psimd' 2024-08-21T00:36:50.2563014Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2606845Z Entering 'third_party/pthreadpool' 2024-08-21T00:36:50.2648283Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2691776Z Entering 'third_party/pybind11' 2024-08-21T00:36:50.2734300Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2777335Z Entering 'third_party/python-peachpy' 2024-08-21T00:36:50.2823927Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2865917Z Entering 'third_party/sleef' 2024-08-21T00:36:50.2908853Z http.https://github.com/.extraheader 2024-08-21T00:36:50.2951218Z Entering 'third_party/tensorpipe' 2024-08-21T00:36:50.2994612Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3036239Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:36:50.3078877Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3121396Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:36:50.3163328Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3209782Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:36:50.3250901Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3294068Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:36:50.3335925Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3377278Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:36:50.3420062Z http.https://github.com/.extraheader 2024-08-21T00:36:50.3577448Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:36:50.3598872Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:36:50.3606915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:36:50.3607427Z ##[endgroup] 2024-08-21T00:36:57.3039811Z Cleaning up orphan processes