2024-08-20T21:40:06.3278203Z Current runner version: '2.319.1' 2024-08-20T21:40:06.3284212Z Runner name: 'i-07cb510f54fcb7b8e' 2024-08-20T21:40:06.3284962Z Runner group name: 'Default' 2024-08-20T21:40:06.3285749Z Machine name: 'ip-10-0-31-108' 2024-08-20T21:40:06.3301270Z Testing runner upgrade compatibility 2024-08-20T21:40:06.4451503Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T21:40:06.4453726Z Actions: read 2024-08-20T21:40:06.4454445Z Attestations: read 2024-08-20T21:40:06.4455030Z Checks: read 2024-08-20T21:40:06.4455493Z Contents: read 2024-08-20T21:40:06.4455959Z Deployments: read 2024-08-20T21:40:06.4456437Z Discussions: read 2024-08-20T21:40:06.4456901Z Issues: read 2024-08-20T21:40:06.4457311Z Metadata: read 2024-08-20T21:40:06.4457771Z Packages: read 2024-08-20T21:40:06.4458238Z Pages: read 2024-08-20T21:40:06.4458659Z PullRequests: read 2024-08-20T21:40:06.4459182Z RepositoryProjects: read 2024-08-20T21:40:06.4459699Z SecurityEvents: read 2024-08-20T21:40:06.4460133Z Statuses: read 2024-08-20T21:40:06.4460782Z ##[endgroup] 2024-08-20T21:40:06.4464154Z Secret source: Actions 2024-08-20T21:40:06.4465034Z Prepare workflow directory 2024-08-20T21:40:06.7822840Z Prepare all required actions 2024-08-20T21:40:06.7989161Z Getting action download info 2024-08-20T21:40:07.0000055Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T21:40:07.3544672Z Download action repository 'pytorch/pytorch@main' (SHA:15b5a0b67fc3f34fb0bf1afa6f91e0c4c2b7fd8d) 2024-08-20T21:40:10.2707586Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T21:40:10.5007532Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T21:40:10.7591029Z Getting action download info 2024-08-20T21:40:10.8999855Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T21:40:11.0577853Z Getting action download info 2024-08-20T21:40:11.1989569Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T21:40:11.3422706Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T21:40:11.3425207Z ##[group] Inputs 2024-08-20T21:40:11.3425651Z build-environment: linux-jammy-py3.8-gcc11-build 2024-08-20T21:40:11.3445494Z test-matrix: {"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]} 2024-08-20T21:40:11.3466832Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:40:11.3467999Z sync-tag: 2024-08-20T21:40:11.3469004Z timeout-minutes: 240 2024-08-20T21:40:11.3469334Z use-gha: 2024-08-20T21:40:11.3469593Z dashboard-tag: 2024-08-20T21:40:11.3469901Z s3-bucket: gha-artifacts 2024-08-20T21:40:11.3470355Z aws-role-to-assume: 2024-08-20T21:40:11.3470664Z ##[endgroup] 2024-08-20T21:40:11.3471649Z Complete job name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:40:11.4240660Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T21:40:11.4374180Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T21:40:11.4384024Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:40:11.4384754Z ##[endgroup] 2024-08-20T21:40:12.9565656Z Runner Type: amz2023.linux.12xlarge 2024-08-20T21:40:12.9566169Z Instance Type: c5.12xlarge 2024-08-20T21:40:12.9566843Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T21:40:12.9567370Z AMI ID: ami-06c68f701d8090592 2024-08-20T21:40:18.2678985Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T21:40:18.2679547Z with: 2024-08-20T21:40:18.2680194Z github-secret: *** 2024-08-20T21:40:18.2681076Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:40:18.2682021Z activate-with-label: false 2024-08-20T21:40:18.2682345Z label: with-ssh 2024-08-20T21:40:18.2682628Z remove-existing-keys: true 2024-08-20T21:40:18.2682954Z fail-silently: true 2024-08-20T21:40:18.2683215Z env: 2024-08-20T21:40:18.2683453Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.2683762Z ##[endgroup] 2024-08-20T21:40:18.3492728Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T21:40:18.3494047Z ciflow reference detected, attempting to extract PR number 2024-08-20T21:40:18.6564032Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T21:40:18.7208064Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T21:40:18.7208735Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T21:40:18.7876612Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T21:40:18.7889501Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T21:40:18.7915324Z Login using: ssh ec2-user@ec2-3-223-125-229.compute-1.amazonaws.com 2024-08-20T21:40:18.7916196Z All testing is done inside the container, to start an interactive session run: 2024-08-20T21:40:18.7917041Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:40:18.8036596Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T21:40:18.8037176Z with: 2024-08-20T21:40:18.8037454Z submodules: recursive 2024-08-20T21:40:18.8037776Z fetch-depth: 0 2024-08-20T21:40:18.8038060Z env: 2024-08-20T21:40:18.8038330Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.8038659Z ##[endgroup] 2024-08-20T21:40:18.8225016Z ##[group]Run retry () { 2024-08-20T21:40:18.8225399Z retry () { 2024-08-20T21:40:18.8225904Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T21:40:18.8226476Z } 2024-08-20T21:40:18.8226768Z echo "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.8227190Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T21:40:18.8227675Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.8228131Z else 2024-08-20T21:40:18.8228460Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.8228895Z fi 2024-08-20T21:40:18.8229252Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.8237944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:40:18.8238436Z env: 2024-08-20T21:40:18.8238695Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.8239002Z NO_SUDO: 2024-08-20T21:40:18.8239256Z ##[endgroup] 2024-08-20T21:40:18.8266409Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.1672969Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T21:40:21.1673356Z with: 2024-08-20T21:40:21.1673649Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:40:21.1674058Z fetch-depth: 0 2024-08-20T21:40:21.1674334Z submodules: recursive 2024-08-20T21:40:21.1674623Z quiet-checkout: true 2024-08-20T21:40:21.1674929Z repository: pytorch/pytorch 2024-08-20T21:40:21.1675379Z token: *** 2024-08-20T21:40:21.1675627Z ssh-strict: true 2024-08-20T21:40:21.1675900Z persist-credentials: true 2024-08-20T21:40:21.1676216Z clean: true 2024-08-20T21:40:21.1676519Z sparse-checkout-cone-mode: true 2024-08-20T21:40:21.1676864Z lfs: false 2024-08-20T21:40:21.1677286Z set-safe-directory: true 2024-08-20T21:40:21.1677589Z env: 2024-08-20T21:40:21.1677824Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:21.1678111Z ##[endgroup] 2024-08-20T21:40:21.2576903Z Syncing repository: pytorch/pytorch 2024-08-20T21:40:21.2578639Z ##[group]Getting Git version info 2024-08-20T21:40:21.2579399Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:40:21.2580262Z [command]/usr/bin/git version 2024-08-20T21:40:21.2580613Z git version 2.40.1 2024-08-20T21:40:21.2581903Z ##[endgroup] 2024-08-20T21:40:21.2593111Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/dadf36ac-e351-4a5a-a4d1-6ca671d86bda' before making global git config changes 2024-08-20T21:40:21.2594401Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T21:40:21.2595573Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.2623353Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:40:21.2626930Z ##[group]Initializing the repository 2024-08-20T21:40:21.2629621Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.2659306Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T21:40:21.2660156Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T21:40:21.2660955Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T21:40:21.2661512Z hint: 2024-08-20T21:40:21.2662008Z hint: git config --global init.defaultBranch 2024-08-20T21:40:21.2662460Z hint: 2024-08-20T21:40:21.2662976Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T21:40:21.2663818Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T21:40:21.2664627Z hint: 2024-08-20T21:40:21.2664939Z hint: git branch -m 2024-08-20T21:40:21.2665697Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T21:40:21.2668783Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T21:40:21.2700371Z ##[endgroup] 2024-08-20T21:40:21.2700923Z ##[group]Disabling automatic garbage collection 2024-08-20T21:40:21.2702027Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T21:40:21.2731203Z ##[endgroup] 2024-08-20T21:40:21.2731710Z ##[group]Setting up auth 2024-08-20T21:40:21.2735232Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T21:40:21.2765372Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T21:40:21.3034452Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T21:40:21.3063587Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T21:40:21.3320411Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:40:21.3370781Z ##[endgroup] 2024-08-20T21:40:21.3371362Z ##[group]Fetching the repository 2024-08-20T21:40:21.3375369Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T21:40:24.2986158Z remote: Enumerating objects: 1008510 2024-08-20T21:40:24.2987092Z remote: Enumerating objects: 1010152, done. 2024-08-20T21:40:24.2988526Z remote: Counting objects: 0% (1/1642) 2024-08-20T21:40:24.2989222Z remote: Counting objects: 1% (17/1642) 2024-08-20T21:40:24.2989764Z remote: Counting objects: 2% (33/1642) 2024-08-20T21:40:24.2990576Z remote: Counting objects: 3% (50/1642) 2024-08-20T21:40:24.2991060Z remote: Counting objects: 4% (66/1642) 2024-08-20T21:40:24.2991625Z remote: Counting objects: 5% (83/1642) 2024-08-20T21:40:24.2992130Z remote: Counting objects: 6% (99/1642) 2024-08-20T21:40:24.2992846Z remote: Counting objects: 7% (115/1642) 2024-08-20T21:40:24.2993576Z remote: Counting objects: 8% (132/1642) 2024-08-20T21:40:24.2994319Z remote: Counting objects: 9% (148/1642) 2024-08-20T21:40:24.2995215Z remote: Counting objects: 10% (165/1642) 2024-08-20T21:40:24.2996113Z remote: Counting objects: 11% (181/1642) 2024-08-20T21:40:24.2996985Z remote: Counting objects: 12% (198/1642) 2024-08-20T21:40:24.2997613Z remote: Counting objects: 13% (214/1642) 2024-08-20T21:40:24.2998498Z remote: Counting objects: 14% (230/1642) 2024-08-20T21:40:24.2999085Z remote: Counting objects: 15% (247/1642) 2024-08-20T21:40:24.2999676Z remote: Counting objects: 16% (263/1642) 2024-08-20T21:40:24.3000194Z remote: Counting objects: 17% (280/1642) 2024-08-20T21:40:24.3000765Z remote: Counting objects: 18% (296/1642) 2024-08-20T21:40:24.3001290Z remote: Counting objects: 19% (312/1642) 2024-08-20T21:40:24.3001819Z remote: Counting objects: 20% (329/1642) 2024-08-20T21:40:24.3002319Z remote: Counting objects: 21% (345/1642) 2024-08-20T21:40:24.3002825Z remote: Counting objects: 22% (362/1642) 2024-08-20T21:40:24.3003304Z remote: Counting objects: 23% (378/1642) 2024-08-20T21:40:24.3003804Z remote: Counting objects: 24% (395/1642) 2024-08-20T21:40:24.3004307Z remote: Counting objects: 25% (411/1642) 2024-08-20T21:40:24.3004802Z remote: Counting objects: 26% (427/1642) 2024-08-20T21:40:24.3005284Z remote: Counting objects: 27% (444/1642) 2024-08-20T21:40:24.3005981Z remote: Counting objects: 28% (460/1642) 2024-08-20T21:40:24.3006496Z remote: Counting objects: 29% (477/1642) 2024-08-20T21:40:24.3007001Z remote: Counting objects: 30% (493/1642) 2024-08-20T21:40:24.3007484Z remote: Counting objects: 31% (510/1642) 2024-08-20T21:40:24.3007988Z remote: Counting objects: 32% (526/1642) 2024-08-20T21:40:24.3008490Z remote: Counting objects: 33% (542/1642) 2024-08-20T21:40:24.3008987Z remote: Counting objects: 34% (559/1642) 2024-08-20T21:40:24.3009468Z remote: Counting objects: 35% (575/1642) 2024-08-20T21:40:24.3010018Z remote: Counting objects: 36% (592/1642) 2024-08-20T21:40:24.3010520Z remote: Counting objects: 37% (608/1642) 2024-08-20T21:40:24.3011014Z remote: Counting objects: 38% (624/1642) 2024-08-20T21:40:24.3011510Z remote: Counting objects: 39% (641/1642) 2024-08-20T21:40:24.3011989Z remote: Counting objects: 40% (657/1642) 2024-08-20T21:40:24.3012495Z remote: Counting objects: 41% (674/1642) 2024-08-20T21:40:24.3013001Z remote: Counting objects: 42% (690/1642) 2024-08-20T21:40:24.3013497Z remote: Counting objects: 43% (707/1642) 2024-08-20T21:40:24.3013976Z remote: Counting objects: 44% (723/1642) 2024-08-20T21:40:24.3014481Z remote: Counting objects: 45% (739/1642) 2024-08-20T21:40:24.3014983Z remote: Counting objects: 46% (756/1642) 2024-08-20T21:40:24.3015481Z remote: Counting objects: 47% (772/1642) 2024-08-20T21:40:24.3015962Z remote: Counting objects: 48% (789/1642) 2024-08-20T21:40:24.3016466Z remote: Counting objects: 49% (805/1642) 2024-08-20T21:40:24.3017078Z remote: Counting objects: 50% (821/1642) 2024-08-20T21:40:24.3017576Z remote: Counting objects: 51% (838/1642) 2024-08-20T21:40:24.3018059Z remote: Counting objects: 52% (854/1642) 2024-08-20T21:40:24.3018657Z remote: Counting objects: 53% (871/1642) 2024-08-20T21:40:24.3019242Z remote: Counting objects: 54% (887/1642) 2024-08-20T21:40:24.3019864Z remote: Counting objects: 55% (904/1642) 2024-08-20T21:40:24.3020345Z remote: Counting objects: 56% (920/1642) 2024-08-20T21:40:24.3020854Z remote: Counting objects: 57% (936/1642) 2024-08-20T21:40:24.3021357Z remote: Counting objects: 58% (953/1642) 2024-08-20T21:40:24.3021858Z remote: Counting objects: 59% (969/1642) 2024-08-20T21:40:24.3022339Z remote: Counting objects: 60% (986/1642) 2024-08-20T21:40:24.3022854Z remote: Counting objects: 61% (1002/1642) 2024-08-20T21:40:24.3023381Z remote: Counting objects: 62% (1019/1642) 2024-08-20T21:40:24.3023917Z remote: Counting objects: 63% (1035/1642) 2024-08-20T21:40:24.3024440Z remote: Counting objects: 64% (1051/1642) 2024-08-20T21:40:24.3024946Z remote: Counting objects: 65% (1068/1642) 2024-08-20T21:40:24.3025477Z remote: Counting objects: 66% (1084/1642) 2024-08-20T21:40:24.3025996Z remote: Counting objects: 67% (1101/1642) 2024-08-20T21:40:24.3026525Z remote: Counting objects: 68% (1117/1642) 2024-08-20T21:40:24.3027026Z remote: Counting objects: 69% (1133/1642) 2024-08-20T21:40:24.3027545Z remote: Counting objects: 70% (1150/1642) 2024-08-20T21:40:24.3028060Z remote: Counting objects: 71% (1166/1642) 2024-08-20T21:40:24.3028576Z remote: Counting objects: 72% (1183/1642) 2024-08-20T21:40:24.3029077Z remote: Counting objects: 73% (1199/1642) 2024-08-20T21:40:24.3029595Z remote: Counting objects: 74% (1216/1642) 2024-08-20T21:40:24.3030113Z remote: Counting objects: 75% (1232/1642) 2024-08-20T21:40:24.3030628Z remote: Counting objects: 76% (1248/1642) 2024-08-20T21:40:24.3031129Z remote: Counting objects: 77% (1265/1642) 2024-08-20T21:40:24.3031641Z remote: Counting objects: 78% (1281/1642) 2024-08-20T21:40:24.3032244Z remote: Counting objects: 79% (1298/1642) 2024-08-20T21:40:24.3032771Z remote: Counting objects: 80% (1314/1642) 2024-08-20T21:40:24.3033269Z remote: Counting objects: 81% (1331/1642) 2024-08-20T21:40:24.3033785Z remote: Counting objects: 82% (1347/1642) 2024-08-20T21:40:24.3034302Z remote: Counting objects: 83% (1363/1642) 2024-08-20T21:40:24.3034822Z remote: Counting objects: 84% (1380/1642) 2024-08-20T21:40:24.3035323Z remote: Counting objects: 85% (1396/1642) 2024-08-20T21:40:24.3035839Z remote: Counting objects: 86% (1413/1642) 2024-08-20T21:40:24.3036357Z remote: Counting objects: 87% (1429/1642) 2024-08-20T21:40:24.3036874Z remote: Counting objects: 88% (1445/1642) 2024-08-20T21:40:24.3037376Z remote: Counting objects: 89% (1462/1642) 2024-08-20T21:40:24.3037892Z remote: Counting objects: 90% (1478/1642) 2024-08-20T21:40:24.3038409Z remote: Counting objects: 91% (1495/1642) 2024-08-20T21:40:24.3038927Z remote: Counting objects: 92% (1511/1642) 2024-08-20T21:40:24.3039433Z remote: Counting objects: 93% (1528/1642) 2024-08-20T21:40:24.3039948Z remote: Counting objects: 94% (1544/1642) 2024-08-20T21:40:24.3040464Z remote: Counting objects: 95% (1560/1642) 2024-08-20T21:40:24.3040963Z remote: Counting objects: 96% (1577/1642) 2024-08-20T21:40:24.3041481Z remote: Counting objects: 97% (1593/1642) 2024-08-20T21:40:24.3041999Z remote: Counting objects: 98% (1610/1642) 2024-08-20T21:40:24.3042513Z remote: Counting objects: 99% (1626/1642) 2024-08-20T21:40:24.3043016Z remote: Counting objects: 100% (1642/1642) 2024-08-20T21:40:24.3043747Z remote: Counting objects: 100% (1642/1642), done. 2024-08-20T21:40:24.3239372Z remote: Compressing objects: 0% (1/835) 2024-08-20T21:40:24.3542209Z remote: Compressing objects: 1% (9/835) 2024-08-20T21:40:24.3958450Z remote: Compressing objects: 2% (17/835) 2024-08-20T21:40:24.4637824Z remote: Compressing objects: 3% (26/835) 2024-08-20T21:40:24.5425982Z remote: Compressing objects: 4% (34/835) 2024-08-20T21:40:24.5921258Z remote: Compressing objects: 5% (42/835) 2024-08-20T21:40:24.6404794Z remote: Compressing objects: 6% (51/835) 2024-08-20T21:40:24.6777582Z remote: Compressing objects: 7% (59/835) 2024-08-20T21:40:24.7127111Z remote: Compressing objects: 8% (67/835) 2024-08-20T21:40:24.7422297Z remote: Compressing objects: 9% (76/835) 2024-08-20T21:40:24.7652668Z remote: Compressing objects: 10% (84/835) 2024-08-20T21:40:24.7876867Z remote: Compressing objects: 11% (92/835) 2024-08-20T21:40:24.8102173Z remote: Compressing objects: 12% (101/835) 2024-08-20T21:40:24.8258936Z remote: Compressing objects: 13% (109/835) 2024-08-20T21:40:24.8372941Z remote: Compressing objects: 14% (117/835) 2024-08-20T21:40:24.8446965Z remote: Compressing objects: 15% (126/835) 2024-08-20T21:40:24.8543531Z remote: Compressing objects: 16% (134/835) 2024-08-20T21:40:24.8635052Z remote: Compressing objects: 17% (142/835) 2024-08-20T21:40:24.8689299Z remote: Compressing objects: 18% (151/835) 2024-08-20T21:40:24.8728342Z remote: Compressing objects: 19% (159/835) 2024-08-20T21:40:24.8768887Z remote: Compressing objects: 20% (167/835) 2024-08-20T21:40:24.8796924Z remote: Compressing objects: 21% (176/835) 2024-08-20T21:40:24.8810883Z remote: Compressing objects: 22% (184/835) 2024-08-20T21:40:24.8811603Z remote: Compressing objects: 23% (193/835) 2024-08-20T21:40:24.8821421Z remote: Compressing objects: 24% (201/835) 2024-08-20T21:40:24.8835454Z remote: Compressing objects: 25% (209/835) 2024-08-20T21:40:24.8836195Z remote: Compressing objects: 26% (218/835) 2024-08-20T21:40:24.8856225Z remote: Compressing objects: 27% (226/835) 2024-08-20T21:40:24.8871484Z remote: Compressing objects: 28% (234/835) 2024-08-20T21:40:24.8894043Z remote: Compressing objects: 29% (243/835) 2024-08-20T21:40:24.8894773Z remote: Compressing objects: 30% (251/835) 2024-08-20T21:40:24.8918328Z remote: Compressing objects: 31% (259/835) 2024-08-20T21:40:24.8925000Z remote: Compressing objects: 32% (268/835) 2024-08-20T21:40:24.8930489Z remote: Compressing objects: 33% (276/835) 2024-08-20T21:40:24.8942799Z remote: Compressing objects: 34% (284/835) 2024-08-20T21:40:24.8994764Z remote: Compressing objects: 35% (293/835) 2024-08-20T21:40:24.8996861Z remote: Compressing objects: 36% (301/835) 2024-08-20T21:40:24.8997628Z remote: Compressing objects: 37% (309/835) 2024-08-20T21:40:24.8998165Z remote: Compressing objects: 38% (318/835) 2024-08-20T21:40:24.8998706Z remote: Compressing objects: 39% (326/835) 2024-08-20T21:40:24.8999246Z remote: Compressing objects: 40% (334/835) 2024-08-20T21:40:24.8999796Z remote: Compressing objects: 41% (343/835) 2024-08-20T21:40:24.9016014Z remote: Compressing objects: 42% (351/835) 2024-08-20T21:40:24.9016608Z remote: Compressing objects: 43% (360/835) 2024-08-20T21:40:24.9024288Z remote: Compressing objects: 44% (368/835) 2024-08-20T21:40:24.9031773Z remote: Compressing objects: 45% (376/835) 2024-08-20T21:40:24.9037770Z remote: Compressing objects: 46% (385/835) 2024-08-20T21:40:24.9041799Z remote: Compressing objects: 47% (393/835) 2024-08-20T21:40:24.9046315Z remote: Compressing objects: 48% (401/835) 2024-08-20T21:40:24.9052216Z remote: Compressing objects: 49% (410/835) 2024-08-20T21:40:24.9056475Z remote: Compressing objects: 50% (418/835) 2024-08-20T21:40:24.9059416Z remote: Compressing objects: 51% (426/835) 2024-08-20T21:40:24.9063474Z remote: Compressing objects: 52% (435/835) 2024-08-20T21:40:24.9067920Z remote: Compressing objects: 53% (443/835) 2024-08-20T21:40:24.9070769Z remote: Compressing objects: 54% (451/835) 2024-08-20T21:40:24.9073365Z remote: Compressing objects: 55% (460/835) 2024-08-20T21:40:24.9077227Z remote: Compressing objects: 56% (468/835) 2024-08-20T21:40:24.9079655Z remote: Compressing objects: 57% (476/835) 2024-08-20T21:40:24.9082180Z remote: Compressing objects: 58% (485/835) 2024-08-20T21:40:24.9083411Z remote: Compressing objects: 59% (493/835) 2024-08-20T21:40:24.9084325Z remote: Compressing objects: 60% (501/835) 2024-08-20T21:40:24.9084916Z remote: Compressing objects: 61% (510/835) 2024-08-20T21:40:24.9086439Z remote: Compressing objects: 62% (518/835) 2024-08-20T21:40:24.9086981Z remote: Compressing objects: 63% (527/835) 2024-08-20T21:40:24.9089043Z remote: Compressing objects: 64% (535/835) 2024-08-20T21:40:24.9093104Z remote: Compressing objects: 65% (543/835) 2024-08-20T21:40:24.9100690Z remote: Compressing objects: 66% (552/835) 2024-08-20T21:40:24.9105238Z remote: Compressing objects: 67% (560/835) 2024-08-20T21:40:24.9109209Z remote: Compressing objects: 68% (568/835) 2024-08-20T21:40:24.9113575Z remote: Compressing objects: 69% (577/835) 2024-08-20T21:40:24.9118338Z remote: Compressing objects: 70% (585/835) 2024-08-20T21:40:24.9125277Z remote: Compressing objects: 71% (593/835) 2024-08-20T21:40:24.9129576Z remote: Compressing objects: 72% (602/835) 2024-08-20T21:40:24.9133660Z remote: Compressing objects: 73% (610/835) 2024-08-20T21:40:24.9139393Z remote: Compressing objects: 74% (618/835) 2024-08-20T21:40:24.9142197Z remote: Compressing objects: 75% (627/835) 2024-08-20T21:40:24.9144578Z remote: Compressing objects: 76% (635/835) 2024-08-20T21:40:24.9148232Z remote: Compressing objects: 77% (643/835) 2024-08-20T21:40:24.9151523Z remote: Compressing objects: 78% (652/835) 2024-08-20T21:40:24.9155200Z remote: Compressing objects: 79% (660/835) 2024-08-20T21:40:24.9156611Z remote: Compressing objects: 80% (668/835) 2024-08-20T21:40:24.9159878Z remote: Compressing objects: 81% (677/835) 2024-08-20T21:40:24.9163068Z remote: Compressing objects: 82% (685/835) 2024-08-20T21:40:24.9165892Z remote: Compressing objects: 83% (694/835) 2024-08-20T21:40:24.9168567Z remote: Compressing objects: 84% (702/835) 2024-08-20T21:40:24.9171008Z remote: Compressing objects: 85% (710/835) 2024-08-20T21:40:24.9173674Z remote: Compressing objects: 86% (719/835) 2024-08-20T21:40:24.9176061Z remote: Compressing objects: 87% (727/835) 2024-08-20T21:40:24.9178749Z remote: Compressing objects: 88% (735/835) 2024-08-20T21:40:24.9181495Z remote: Compressing objects: 89% (744/835) 2024-08-20T21:40:24.9184574Z remote: Compressing objects: 90% (752/835) 2024-08-20T21:40:24.9187850Z remote: Compressing objects: 91% (760/835) 2024-08-20T21:40:24.9190330Z remote: Compressing objects: 92% (769/835) 2024-08-20T21:40:24.9190844Z remote: Compressing objects: 93% (777/835) 2024-08-20T21:40:24.9192956Z remote: Compressing objects: 94% (785/835) 2024-08-20T21:40:24.9195733Z remote: Compressing objects: 95% (794/835) 2024-08-20T21:40:24.9196233Z remote: Compressing objects: 96% (802/835) 2024-08-20T21:40:24.9198494Z remote: Compressing objects: 97% (810/835) 2024-08-20T21:40:24.9201134Z remote: Compressing objects: 98% (819/835) 2024-08-20T21:40:24.9201645Z remote: Compressing objects: 99% (827/835) 2024-08-20T21:40:24.9203829Z remote: Compressing objects: 100% (835/835) 2024-08-20T21:40:24.9204406Z remote: Compressing objects: 100% (835/835), done. 2024-08-20T21:40:44.9159777Z remote: Total 1010152 (delta 1073), reused 1263 (delta 804), pack-reused 1008510 (from 1) 2024-08-20T21:41:06.0485445Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T21:41:06.0517247Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:41:06.0519920Z ##[endgroup] 2024-08-20T21:41:06.0520494Z ##[group]Determining the checkout info 2024-08-20T21:41:06.0521236Z ##[endgroup] 2024-08-20T21:41:06.0521733Z ##[group]Checking out the ref 2024-08-20T21:41:06.0524710Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:41:07.3677805Z ##[endgroup] 2024-08-20T21:41:07.3678406Z ##[group]Setting up auth for fetching submodules 2024-08-20T21:41:07.3679531Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:41:07.3733554Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T21:41:07.3761150Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T21:41:07.3790773Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T21:41:07.3818828Z ##[endgroup] 2024-08-20T21:41:07.3819359Z ##[group]Fetching submodules 2024-08-20T21:41:07.3820707Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T21:41:07.4108069Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T21:41:07.4390810Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T21:41:07.4392300Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T21:41:07.4393895Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T21:41:07.4396187Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T21:41:07.4399137Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T21:41:07.4401193Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T21:41:07.4403947Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T21:41:07.4406624Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T21:41:07.4409245Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T21:41:07.4412236Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T21:41:07.4415051Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T21:41:07.4418040Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T21:41:07.4421369Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T21:41:07.4424695Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T21:41:07.4427764Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T21:41:07.4431456Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:41:07.4436183Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T21:41:07.4440242Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T21:41:07.4443762Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T21:41:07.4447881Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T21:41:07.4451670Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T21:41:07.4456055Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T21:41:07.4460204Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T21:41:07.4464337Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T21:41:07.4468441Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T21:41:07.4473087Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T21:41:07.4477202Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T21:41:07.4481742Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T21:41:07.4486276Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T21:41:07.4491040Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T21:41:07.4495699Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T21:41:07.4500859Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T21:41:07.4505571Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T21:41:07.4512032Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T21:41:07.4538735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T21:41:07.7279150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T21:41:07.8962945Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T21:41:08.0669379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T21:41:08.3032045Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T21:41:09.9640151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T21:41:18.5958190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T21:41:19.0058674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T21:41:19.3961244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T21:41:19.9526691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T21:41:21.0169888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T21:41:22.7561975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T21:41:27.8678879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T21:41:29.0751926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T21:41:30.5939986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T21:41:31.5851560Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T21:41:31.9633974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T21:41:32.2834887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T21:41:33.1574113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T21:41:33.4846090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T21:41:33.7514506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T21:41:35.2789692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T21:41:35.9084853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T21:41:36.5420596Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T21:41:42.4855166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T21:41:44.1983137Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T21:41:47.1266614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T21:41:47.3479374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T21:41:54.8362784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T21:41:54.9972901Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T21:41:55.1798024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T21:41:55.9169614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T21:41:56.2182496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T21:41:56.8113633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T21:41:57.1868978Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T21:41:57.1982819Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T21:41:57.2067516Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T21:41:57.2292184Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T21:41:57.2624359Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T21:41:58.1499879Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T21:41:58.1721944Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T21:41:58.2140030Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T21:41:58.3038735Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T21:41:58.3342555Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T21:41:58.7983343Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T21:41:59.0323033Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T21:41:59.1106829Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T21:41:59.1124484Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:41:59.1126962Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:41:59.1128550Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:41:59.1130127Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:41:59.1131967Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:41:59.1158130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T21:42:00.0652636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T21:42:00.6348629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T21:42:02.4071043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T21:42:03.2513247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T21:42:03.5672801Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T21:42:03.6563148Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T21:42:04.0279492Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T21:42:04.0860584Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T21:42:04.0980587Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T21:42:04.2050051Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T21:42:04.2423730Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T21:42:04.2782063Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T21:42:04.3014518Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T21:42:04.3436434Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:42:04.3554007Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T21:42:04.3570973Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:04.3593814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T21:42:15.1345093Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T21:42:15.1519848Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T21:42:15.2336479Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T21:42:15.2353742Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:15.2355779Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:15.2357570Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:15.2381848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T21:42:15.7343452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T21:42:16.7369087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T21:42:17.6986193Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T21:42:17.7003010Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:17.7004817Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:17.7006606Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:17.7008429Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:17.7010346Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:17.7013166Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:17.7015803Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:17.7018716Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:17.7046695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T21:42:18.4557240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T21:42:18.8028111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T21:42:19.7356818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T21:42:19.9877987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T21:42:20.4794501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T21:42:21.3530061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T21:42:26.2656138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T21:42:26.6739180Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T21:42:26.6907591Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T21:42:26.7248022Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T21:42:26.7370391Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T21:42:26.7384295Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:26.7409996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T21:42:27.0278185Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T21:42:27.0452131Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T21:42:27.0821614Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T21:42:27.1782128Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T21:42:27.1935021Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T21:42:27.2316605Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T21:42:27.2865475Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T21:42:27.3214674Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T21:42:27.3429681Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T21:42:27.4403079Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T21:42:27.7657756Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T21:42:27.7691710Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:27.7693396Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:27.7720040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T21:42:28.1794252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T21:42:28.9246468Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T21:42:28.9559613Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T21:42:29.0184739Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T21:42:29.0203609Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:29.0205405Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:29.0207088Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:29.0208776Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:29.0211008Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:29.0213481Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:29.0216213Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:29.0218862Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:29.0253329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T21:42:29.3925353Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T21:42:30.2728438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T21:42:30.6015582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T21:42:36.5779952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T21:42:36.8532363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T21:42:37.0388086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T21:42:37.3135801Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T21:42:42.7278347Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T21:42:42.7659667Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T21:42:42.7803928Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T21:42:42.8725541Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T21:42:42.8848870Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T21:42:42.8980796Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T21:42:42.9122113Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T21:42:42.9137491Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:42.9139572Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:42.9161795Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T21:42:44.4257770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T21:42:45.5317427Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T21:42:45.5743668Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:42:45.9997964Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T21:42:46.0116097Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T21:42:46.2573604Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T21:42:46.2595097Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:46.2597097Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:46.2621567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T21:42:46.6855847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T21:42:47.5529349Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T21:42:47.6182059Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T21:42:47.6266953Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T21:42:47.6381018Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T21:42:47.6714226Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T21:42:47.6976305Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T21:42:47.7358510Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T21:42:47.7602476Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T21:42:47.7619020Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:47.7620712Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:47.7622470Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:47.7624077Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:47.7649917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T21:42:48.6759221Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T21:42:48.9082602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T21:42:49.9185968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T21:42:50.6985900Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T21:42:50.7127655Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T21:42:50.7746669Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T21:42:50.8009173Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T21:42:50.8022392Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:50.8048442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T21:42:50.9980120Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T21:42:51.0017851Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T21:42:51.0321134Z Entering 'android/libs/fbjni' 2024-08-20T21:42:51.0360252Z Entering 'third_party/FP16' 2024-08-20T21:42:51.0401530Z Entering 'third_party/FXdiv' 2024-08-20T21:42:51.0441929Z Entering 'third_party/NNPACK' 2024-08-20T21:42:51.0482054Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:51.0521935Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:51.0579928Z Entering 'third_party/benchmark' 2024-08-20T21:42:51.0619903Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:51.0657855Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:51.0698921Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:51.0739631Z Entering 'third_party/cutlass' 2024-08-20T21:42:51.0790491Z Entering 'third_party/eigen' 2024-08-20T21:42:51.0833994Z Entering 'third_party/fbgemm' 2024-08-20T21:42:51.0873958Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:51.0909264Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:51.0947187Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:51.0993551Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:51.1033130Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:51.1070206Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:51.1115880Z Entering 'third_party/fmt' 2024-08-20T21:42:51.1154603Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:51.1202050Z Entering 'third_party/gloo' 2024-08-20T21:42:51.1240812Z Entering 'third_party/googletest' 2024-08-20T21:42:51.1279927Z Entering 'third_party/ideep' 2024-08-20T21:42:51.1317453Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:51.1359872Z Entering 'third_party/ittapi' 2024-08-20T21:42:51.1399441Z Entering 'third_party/kineto' 2024-08-20T21:42:51.1438313Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:51.1476896Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:51.1517579Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:51.1556281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:51.1595775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:51.1642631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:51.1682845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:51.1720821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:51.1757900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:51.1797453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:51.1846170Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:51.1883957Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:51.1924069Z Entering 'third_party/mimalloc' 2024-08-20T21:42:51.1960189Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:51.2000418Z Entering 'third_party/nlohmann' 2024-08-20T21:42:51.2039604Z Entering 'third_party/onnx' 2024-08-20T21:42:51.2092500Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:51.2131091Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:51.2173211Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:51.2213702Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:51.2251188Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:51.2289443Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:51.2325713Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:51.2361820Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:51.2401089Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:51.2438282Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:51.2476128Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:51.2516944Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:51.2560464Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:51.2619228Z Entering 'third_party/pocketfft' 2024-08-20T21:42:51.2656262Z Entering 'third_party/protobuf' 2024-08-20T21:42:51.2697958Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:51.2739497Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:51.2779445Z Entering 'third_party/psimd' 2024-08-20T21:42:51.2817995Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:51.2855405Z Entering 'third_party/pybind11' 2024-08-20T21:42:51.2894740Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:51.2933670Z Entering 'third_party/sleef' 2024-08-20T21:42:51.2973382Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:51.3013073Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:51.3050994Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:51.3089972Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:51.3133043Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:51.3171290Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:51.3224585Z ##[endgroup] 2024-08-20T21:42:51.3226396Z ##[group]Persisting credentials for submodules 2024-08-20T21:42:51.3229100Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T21:42:51.3510275Z Entering 'android/libs/fbjni' 2024-08-20T21:42:51.3559519Z Entering 'third_party/FP16' 2024-08-20T21:42:51.3614569Z Entering 'third_party/FXdiv' 2024-08-20T21:42:51.3663174Z Entering 'third_party/NNPACK' 2024-08-20T21:42:51.3716407Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:51.3764471Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:51.3832195Z Entering 'third_party/benchmark' 2024-08-20T21:42:51.3882381Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:51.3932567Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:51.3983819Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:51.4035584Z Entering 'third_party/cutlass' 2024-08-20T21:42:51.4092899Z Entering 'third_party/eigen' 2024-08-20T21:42:51.4143354Z Entering 'third_party/fbgemm' 2024-08-20T21:42:51.4195911Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:51.4244500Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:51.4293710Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:51.4347249Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:51.4397636Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:51.4449537Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:51.4510807Z Entering 'third_party/fmt' 2024-08-20T21:42:51.4563043Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:51.4615694Z Entering 'third_party/gloo' 2024-08-20T21:42:51.4664745Z Entering 'third_party/googletest' 2024-08-20T21:42:51.4716842Z Entering 'third_party/ideep' 2024-08-20T21:42:51.4765378Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:51.4823055Z Entering 'third_party/ittapi' 2024-08-20T21:42:51.4878525Z Entering 'third_party/kineto' 2024-08-20T21:42:51.4928935Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:51.4981696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:51.5042645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:51.5095370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:51.5154084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:51.5208281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:51.5257914Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:51.5308142Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:51.5357002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:51.5408179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:51.5462385Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:51.5516618Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:51.5573950Z Entering 'third_party/mimalloc' 2024-08-20T21:42:51.5622005Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:51.5673091Z Entering 'third_party/nlohmann' 2024-08-20T21:42:51.5724195Z Entering 'third_party/onnx' 2024-08-20T21:42:51.5789508Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:51.5837591Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:51.5890297Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:51.5940347Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:51.5995002Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:51.6054105Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:51.6109993Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:51.6159317Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:51.6213831Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:51.6262381Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:51.6315204Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:51.6365965Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:51.6422031Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:51.6494956Z Entering 'third_party/pocketfft' 2024-08-20T21:42:51.6543004Z Entering 'third_party/protobuf' 2024-08-20T21:42:51.6597441Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:51.6647312Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:51.6699788Z Entering 'third_party/psimd' 2024-08-20T21:42:51.6751475Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:51.6801531Z Entering 'third_party/pybind11' 2024-08-20T21:42:51.6852136Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:51.6901018Z Entering 'third_party/sleef' 2024-08-20T21:42:51.6953125Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:51.7003073Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:51.7058372Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:51.7108812Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:51.7157091Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:51.7206727Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:51.7273969Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T21:42:51.7556315Z Entering 'android/libs/fbjni' 2024-08-20T21:42:51.7602653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T21:42:51.7618809Z Entering 'third_party/FP16' 2024-08-20T21:42:51.7663043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T21:42:51.7680722Z Entering 'third_party/FXdiv' 2024-08-20T21:42:51.7729242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T21:42:51.7742826Z Entering 'third_party/NNPACK' 2024-08-20T21:42:51.7788368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T21:42:51.7805711Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:51.7850905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T21:42:51.7864741Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:51.7909503Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T21:42:51.7940535Z Entering 'third_party/benchmark' 2024-08-20T21:42:51.7990788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:51.8008578Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:51.8053596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T21:42:51.8067551Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:51.8116216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:42:51.8131721Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:51.8178747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T21:42:51.8194879Z Entering 'third_party/cutlass' 2024-08-20T21:42:51.8241983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:42:51.8262980Z Entering 'third_party/eigen' 2024-08-20T21:42:51.8309991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T21:42:51.8330568Z Entering 'third_party/fbgemm' 2024-08-20T21:42:51.8378532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T21:42:51.8393663Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:51.8444041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T21:42:51.8458138Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:51.8509933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:42:51.8526439Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:51.8573639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:42:51.8596070Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:51.8641961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:51.8656637Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:51.8709244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T21:42:51.8727840Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:51.8774452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T21:42:51.8793607Z Entering 'third_party/fmt' 2024-08-20T21:42:51.8839176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:42:51.8854994Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:51.8901005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T21:42:51.8918112Z Entering 'third_party/gloo' 2024-08-20T21:42:51.8963774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T21:42:51.8980471Z Entering 'third_party/googletest' 2024-08-20T21:42:51.9026152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:51.9043415Z Entering 'third_party/ideep' 2024-08-20T21:42:51.9089660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T21:42:51.9102090Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:51.9149963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T21:42:51.9174137Z Entering 'third_party/ittapi' 2024-08-20T21:42:51.9219348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T21:42:51.9235622Z Entering 'third_party/kineto' 2024-08-20T21:42:51.9282843Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T21:42:51.9298016Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:51.9342304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T21:42:51.9355895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:51.9404017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T21:42:51.9420322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:51.9465321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T21:42:51.9481670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:51.9529891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:42:51.9543357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:51.9591090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T21:42:51.9607441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:51.9654038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T21:42:51.9668785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:51.9716654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T21:42:51.9732676Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:51.9780251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:51.9796263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:51.9844228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T21:42:51.9859030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:51.9904863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T21:42:51.9923143Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:51.9969556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T21:42:51.9983284Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.0028794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.0047901Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.0092973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T21:42:52.0106711Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.0154445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T21:42:52.0170478Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.0214535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T21:42:52.0228680Z Entering 'third_party/onnx' 2024-08-20T21:42:52.0276213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T21:42:52.0305265Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.0351547Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.0368548Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.0416668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.0435970Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.0484257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T21:42:52.0500363Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.0546111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.0560274Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.0612133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.0625402Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.0673505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T21:42:52.0688575Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.0732931Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T21:42:52.0748182Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.0797318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T21:42:52.0813162Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.0858320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T21:42:52.0874285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.0921147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T21:42:52.0935393Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.0981964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T21:42:52.1000442Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:52.1048099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T21:42:52.1063303Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:52.1110961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T21:42:52.1143479Z Entering 'third_party/pocketfft' 2024-08-20T21:42:52.1190378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T21:42:52.1206419Z Entering 'third_party/protobuf' 2024-08-20T21:42:52.1252723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T21:42:52.1269102Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:52.1316165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.1331380Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:52.1378409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.1396978Z Entering 'third_party/psimd' 2024-08-20T21:42:52.1443655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T21:42:52.1457268Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:52.1503105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T21:42:52.1520086Z Entering 'third_party/pybind11' 2024-08-20T21:42:52.1564760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.1580113Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:52.1623789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T21:42:52.1640638Z Entering 'third_party/sleef' 2024-08-20T21:42:52.1690916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T21:42:52.1704352Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:52.1750672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T21:42:52.1765061Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:52.1811496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.1824892Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:52.1869499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T21:42:52.1885174Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:52.1931239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T21:42:52.1945342Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:52.1992130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.2008871Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:52.2055149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T21:42:52.2519952Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T21:42:52.2810716Z Entering 'android/libs/fbjni' 2024-08-20T21:42:52.2853515Z Entering 'third_party/FP16' 2024-08-20T21:42:52.2900524Z Entering 'third_party/FXdiv' 2024-08-20T21:42:52.2941135Z Entering 'third_party/NNPACK' 2024-08-20T21:42:52.2980353Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:52.3019265Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:52.3074794Z Entering 'third_party/benchmark' 2024-08-20T21:42:52.3114836Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:52.3154828Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:52.3199165Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:52.3237180Z Entering 'third_party/cutlass' 2024-08-20T21:42:52.3283895Z Entering 'third_party/eigen' 2024-08-20T21:42:52.3323639Z Entering 'third_party/fbgemm' 2024-08-20T21:42:52.3364448Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:52.3403103Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:52.3441325Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:52.3485068Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:52.3523409Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:52.3561288Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:52.3602978Z Entering 'third_party/fmt' 2024-08-20T21:42:52.3642358Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:52.3681521Z Entering 'third_party/gloo' 2024-08-20T21:42:52.3720584Z Entering 'third_party/googletest' 2024-08-20T21:42:52.3757806Z Entering 'third_party/ideep' 2024-08-20T21:42:52.3796594Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:52.3848376Z Entering 'third_party/ittapi' 2024-08-20T21:42:52.3888122Z Entering 'third_party/kineto' 2024-08-20T21:42:52.3926680Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:52.3966568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:52.4013231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:52.4052494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:52.4091903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:52.4133456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:52.4175275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:52.4214178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:52.4252332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:52.4292223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:52.4332668Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:52.4371887Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.4412174Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.4451651Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.4492411Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.4532110Z Entering 'third_party/onnx' 2024-08-20T21:42:52.4584879Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.4630702Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.4672491Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.4714449Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.4752553Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.4789916Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.4826854Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.4865817Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.4903401Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.4941683Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.4979529Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.5020164Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:52.5059092Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:52.5118603Z Entering 'third_party/pocketfft' 2024-08-20T21:42:52.5156325Z Entering 'third_party/protobuf' 2024-08-20T21:42:52.5197019Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:52.5235125Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:52.5275764Z Entering 'third_party/psimd' 2024-08-20T21:42:52.5315517Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:52.5354985Z Entering 'third_party/pybind11' 2024-08-20T21:42:52.5394647Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:52.5432666Z Entering 'third_party/sleef' 2024-08-20T21:42:52.5469631Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:52.5508725Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:52.5545777Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:52.5583914Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:52.5621273Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:52.5657978Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:52.5713059Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T21:42:52.6005136Z Entering 'android/libs/fbjni' 2024-08-20T21:42:52.6043071Z Entering 'third_party/FP16' 2024-08-20T21:42:52.6082280Z Entering 'third_party/FXdiv' 2024-08-20T21:42:52.6120448Z Entering 'third_party/NNPACK' 2024-08-20T21:42:52.6158464Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:52.6198189Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:52.6251841Z Entering 'third_party/benchmark' 2024-08-20T21:42:52.6292347Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:52.6331673Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:52.6371957Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:52.6414324Z Entering 'third_party/cutlass' 2024-08-20T21:42:52.6462260Z Entering 'third_party/eigen' 2024-08-20T21:42:52.6502855Z Entering 'third_party/fbgemm' 2024-08-20T21:42:52.6541263Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:52.6579824Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:52.6620834Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:52.6663477Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:52.6701482Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:52.6740826Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:52.6781083Z Entering 'third_party/fmt' 2024-08-20T21:42:52.6820527Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:52.6858300Z Entering 'third_party/gloo' 2024-08-20T21:42:52.6899102Z Entering 'third_party/googletest' 2024-08-20T21:42:52.6938262Z Entering 'third_party/ideep' 2024-08-20T21:42:52.6976336Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:52.7020753Z Entering 'third_party/ittapi' 2024-08-20T21:42:52.7058530Z Entering 'third_party/kineto' 2024-08-20T21:42:52.7098016Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:52.7140992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:52.7182081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:52.7220738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:52.7262305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:52.7300514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:52.7342550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:52.7381436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:52.7428900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:52.7468180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:52.7508011Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:52.7545262Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.7589348Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.7631199Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.7668654Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.7707623Z Entering 'third_party/onnx' 2024-08-20T21:42:52.7760396Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.7800778Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.7841599Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.7881625Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.7919027Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.7956111Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.7993838Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.8034706Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.8072251Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.8108597Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.8149087Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.8190935Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:52.8229093Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:52.8288307Z Entering 'third_party/pocketfft' 2024-08-20T21:42:52.8328054Z Entering 'third_party/protobuf' 2024-08-20T21:42:52.8374495Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:52.8412284Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:52.8453611Z Entering 'third_party/psimd' 2024-08-20T21:42:52.8493605Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:52.8533163Z Entering 'third_party/pybind11' 2024-08-20T21:42:52.8572658Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:52.8615819Z Entering 'third_party/sleef' 2024-08-20T21:42:52.8655007Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:52.8695256Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:52.8733036Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:52.8771175Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:52.8809514Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:52.8847380Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:52.8899072Z ##[endgroup] 2024-08-20T21:42:52.8933369Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T21:42:52.8959131Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T21:42:52.9149854Z Prepare all required actions 2024-08-20T21:42:52.9150325Z Getting action download info 2024-08-20T21:42:53.3012511Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T21:42:53.3012907Z env: 2024-08-20T21:42:53.3013172Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.3013507Z ##[endgroup] 2024-08-20T21:42:53.3075007Z ##[group]Run set -euo pipefail 2024-08-20T21:42:53.3075661Z set -euo pipefail 2024-08-20T21:42:53.3076036Z function get_ec2_metadata() { 2024-08-20T21:42:53.3076579Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T21:42:53.3077505Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T21:42:53.3078412Z  category=$1 2024-08-20T21:42:53.3078912Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T21:42:53.3079541Z  runner_name_str=i-07cb510f54fcb7b8e 2024-08-20T21:42:53.3080173Z  if [[ -f /.inarc ]]; then 2024-08-20T21:42:53.3080616Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T21:42:53.3081150Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T21:42:53.3081798Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T21:42:53.3082375Z  else 2024-08-20T21:42:53.3082816Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T21:42:53.3083357Z  fi 2024-08-20T21:42:53.3083620Z } 2024-08-20T21:42:53.3083948Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T21:42:53.3084479Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T21:42:53.3085092Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T21:42:53.3085626Z echo "system info $(uname -a)" 2024-08-20T21:42:53.3093619Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.3094072Z env: 2024-08-20T21:42:53.3094333Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.3094659Z ##[endgroup] 2024-08-20T21:42:53.3195686Z ami-id: ami-06c68f701d8090592 2024-08-20T21:42:53.3264895Z instance-id: i-07cb510f54fcb7b8e 2024-08-20T21:42:53.3324911Z instance-type: c5.12xlarge 2024-08-20T21:42:53.3336051Z system info Linux ip-10-0-31-108.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T21:42:53.3358013Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:42:53.3358959Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:42:53.3364806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.3365304Z env: 2024-08-20T21:42:53.3365574Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.3365899Z ##[endgroup] 2024-08-20T21:42:53.3432555Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T21:42:53.3433128Z if systemctl is-active --quiet docker; then 2024-08-20T21:42:53.3433630Z  echo "Docker daemon is running..."; 2024-08-20T21:42:53.3434049Z else 2024-08-20T21:42:53.3434489Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T21:42:53.3435035Z fi 2024-08-20T21:42:53.3440098Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.3440561Z env: 2024-08-20T21:42:53.3440795Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.3441111Z ##[endgroup] 2024-08-20T21:42:53.3513763Z Docker daemon is running... 2024-08-20T21:42:53.3559025Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:42:53.3559587Z with: 2024-08-20T21:42:53.3559842Z shell: bash 2024-08-20T21:42:53.3560116Z timeout_minutes: 5 2024-08-20T21:42:53.3560430Z max_attempts: 3 2024-08-20T21:42:53.3560731Z retry_wait_seconds: 30 2024-08-20T21:42:53.3564023Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T21:42:53.3567446Z polling_interval_seconds: 1 2024-08-20T21:42:53.3567790Z warning_on_retry: true 2024-08-20T21:42:53.3568108Z continue_on_error: false 2024-08-20T21:42:53.3568409Z env: 2024-08-20T21:42:53.3568634Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.3568953Z AWS_RETRY_MODE: standard 2024-08-20T21:42:53.3569264Z AWS_MAX_ATTEMPTS: 5 2024-08-20T21:42:53.3569557Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T21:42:53.3570076Z ##[endgroup] 2024-08-20T21:42:54.4346642Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:54.4347482Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:54.4348362Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:54.4348892Z 2024-08-20T21:42:54.4349021Z Login Succeeded 2024-08-20T21:42:55.4095009Z Command completed after 1 attempt(s). 2024-08-20T21:42:55.4151584Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.4152307Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.4153241Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.4161085Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.4161545Z env: 2024-08-20T21:42:55.4161805Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.4162134Z ##[endgroup] 2024-08-20T21:42:55.4247814Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:42:55.4248579Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:42:55.4249163Z # shellcheck disable=SC2046 2024-08-20T21:42:55.4249613Z docker stop $(docker ps -q) || true 2024-08-20T21:42:55.4250065Z # Prune all of the docker images 2024-08-20T21:42:55.4250512Z docker system prune -af 2024-08-20T21:42:55.4256752Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.4257265Z env: 2024-08-20T21:42:55.4257544Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.4257894Z ##[endgroup] 2024-08-20T21:42:55.4546173Z "docker stop" requires at least 1 argument. 2024-08-20T21:42:55.4546891Z See 'docker stop --help'. 2024-08-20T21:42:55.4547142Z 2024-08-20T21:42:55.4547360Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T21:42:55.4547751Z 2024-08-20T21:42:55.4548256Z Stop one or more running containers 2024-08-20T21:42:55.4690404Z Total reclaimed space: 0B 2024-08-20T21:42:55.4723577Z ##[group]Run set +e 2024-08-20T21:42:55.4723922Z set +e 2024-08-20T21:42:55.4724202Z set -x 2024-08-20T21:42:55.4724476Z  2024-08-20T21:42:55.4724847Z PT_DOMAIN=download.pytorch.org 2024-08-20T21:42:55.4725828Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T21:42:55.4727030Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T21:42:55.4727959Z # one is returned at random 2024-08-20T21:42:55.4728541Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T21:42:55.4814605Z  2024-08-20T21:42:55.4814901Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:42:55.4815670Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T21:42:55.4817749Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T21:42:55.4818312Z  2024-08-20T21:42:55.4818733Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:42:55.4819455Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T21:42:55.4819964Z  exit 1 2024-08-20T21:42:55.4820265Z  fi 2024-08-20T21:42:55.4820534Z fi 2024-08-20T21:42:55.4820950Z  2024-08-20T21:42:55.4821280Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T21:42:55.4821789Z  # Clean up any old records first 2024-08-20T21:42:55.4822292Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T21:42:55.4822739Z fi 2024-08-20T21:42:55.4822984Z  2024-08-20T21:42:55.4823399Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T21:42:55.4823937Z cat /etc/hosts 2024-08-20T21:42:55.4830073Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.4830555Z env: 2024-08-20T21:42:55.4830831Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.4831280Z ##[endgroup] 2024-08-20T21:42:55.4855513Z + PT_DOMAIN=download.pytorch.org 2024-08-20T21:42:55.4860674Z ++ dig -4 +short download.pytorch.org 2024-08-20T21:42:55.4861243Z ++ tail -n1 2024-08-20T21:42:55.5100265Z + RESOLVED_IP=18.165.98.47 2024-08-20T21:42:55.5100769Z + '[' -z 18.165.98.47 ']' 2024-08-20T21:42:55.5101225Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T21:42:55.5110854Z 108.138.64.61 download.pytorch.org 2024-08-20T21:42:55.5112995Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T21:42:55.7732590Z + echo '18.165.98.47 download.pytorch.org' 2024-08-20T21:42:55.7733582Z + sudo tee -a /etc/hosts 2024-08-20T21:42:55.8106553Z 18.165.98.47 download.pytorch.org 2024-08-20T21:42:55.8118745Z + cat /etc/hosts 2024-08-20T21:42:55.8128742Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T21:42:55.8140083Z ::1 localhost6 localhost6.localdomain6 2024-08-20T21:42:55.8140759Z 18.165.98.47 download.pytorch.org 2024-08-20T21:42:55.8295721Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T21:42:55.8296283Z with: 2024-08-20T21:42:55.8297250Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8298359Z docker-build-dir: .ci/docker 2024-08-20T21:42:55.8298804Z working-directory: . 2024-08-20T21:42:55.8299414Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:55.8300002Z force-push: false 2024-08-20T21:42:55.8300284Z env: 2024-08-20T21:42:55.8300540Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.8300864Z ##[endgroup] 2024-08-20T21:42:55.8317820Z ##[group]Run set -ex 2024-08-20T21:42:55.8318150Z set -ex 2024-08-20T21:42:55.8318400Z  2024-08-20T21:42:55.8318909Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T21:42:55.8319873Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T21:42:55.8320611Z # job could then download the pre-built image as usual 2024-08-20T21:42:55.8321307Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T21:42:55.8321948Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8322533Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8323045Z  2024-08-20T21:42:55.8323497Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T21:42:55.8324074Z  exit 0 2024-08-20T21:42:55.8324340Z else 2024-08-20T21:42:55.8324662Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8325055Z fi 2024-08-20T21:42:55.8325292Z  2024-08-20T21:42:55.8325712Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T21:42:55.8326509Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T21:42:55.8327224Z  # use it as it is, but first let's extract the tag 2024-08-20T21:42:55.8327872Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T21:42:55.8328649Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8329295Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8329800Z else 2024-08-20T21:42:55.8330187Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T21:42:55.8330808Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8331651Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8332346Z fi 2024-08-20T21:42:55.8339419Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.8340006Z env: 2024-08-20T21:42:55.8340266Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.8340595Z REPO_NAME: pytorch 2024-08-20T21:42:55.8341685Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8342867Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:42:55.8343359Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:55.8343870Z ##[endgroup] 2024-08-20T21:42:55.8368308Z + [[ ! -d .ci/docker ]] 2024-08-20T21:42:55.8368706Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T21:42:55.8369073Z + echo skip=false 2024-08-20T21:42:55.8370863Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T21:42:55.8377116Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8378278Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T21:42:55.8431618Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8432606Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8434166Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8459000Z ##[group]Run set +e 2024-08-20T21:42:55.8459358Z set +e 2024-08-20T21:42:55.8459641Z set -x 2024-08-20T21:42:55.8459901Z  2024-08-20T21:42:55.8460162Z login() { 2024-08-20T21:42:55.8460827Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:42:55.8461552Z } 2024-08-20T21:42:55.8461799Z  2024-08-20T21:42:55.8462055Z retry () { 2024-08-20T21:42:55.8462422Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:42:55.8462853Z } 2024-08-20T21:42:55.8463094Z  2024-08-20T21:42:55.8463393Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:42:55.8463806Z  2024-08-20T21:42:55.8464256Z # Check if image already exists, if it does then skip building it 2024-08-20T21:42:55.8464961Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T21:42:55.8465464Z  exit 0 2024-08-20T21:42:55.8465747Z fi 2024-08-20T21:42:55.8465991Z  2024-08-20T21:42:55.8466486Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T21:42:55.8467327Z # be empty. The default action would be to continue rebuild the image 2024-08-20T21:42:55.8468069Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T21:42:55.8468731Z  # if we're on the base branch then use the parent commit 2024-08-20T21:42:55.8469317Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T21:42:55.8469737Z else 2024-08-20T21:42:55.8470196Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T21:42:55.8471064Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T21:42:55.8471526Z fi 2024-08-20T21:42:55.8471770Z  2024-08-20T21:42:55.8472050Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T21:42:55.8472497Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8472914Z  2024-08-20T21:42:55.8473520Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T21:42:55.8474226Z  exit 0 2024-08-20T21:42:55.8474478Z fi 2024-08-20T21:42:55.8474713Z  2024-08-20T21:42:55.8475269Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T21:42:55.8476233Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T21:42:55.8477010Z  exit 1 2024-08-20T21:42:55.8477282Z fi 2024-08-20T21:42:55.8477532Z  2024-08-20T21:42:55.8478000Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T21:42:55.8478931Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T21:42:55.8479790Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T21:42:55.8480750Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T21:42:55.8481844Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T21:42:55.8482477Z fi 2024-08-20T21:42:55.8482801Z  2024-08-20T21:42:55.8483135Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:55.8488896Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.8489391Z env: 2024-08-20T21:42:55.8489642Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.8489988Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:42:55.8490447Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:42:55.8491685Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8492883Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:55.8493485Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:55.8493993Z ##[endgroup] 2024-08-20T21:42:55.8515763Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:55.8516418Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:55.8518716Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:42:55.8519888Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.3461041Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:56.3461863Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:56.3462898Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:56.3463530Z 2024-08-20T21:42:56.3463637Z Login Succeeded 2024-08-20T21:42:56.3476679Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.5423538Z { 2024-08-20T21:42:56.5423864Z "schemaVersion": 2, 2024-08-20T21:42:56.5424423Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T21:42:56.5425040Z "config": { 2024-08-20T21:42:56.5425515Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T21:42:56.5426072Z "size": 43916, 2024-08-20T21:42:56.5426601Z "digest": "sha256:822356886a68936cce34917f5ffc60d864918fab7594bb08c26bf1816dd6110d" 2024-08-20T21:42:56.5427227Z }, 2024-08-20T21:42:56.5427484Z "layers": [ 2024-08-20T21:42:56.5427732Z { 2024-08-20T21:42:56.5428508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5429082Z "size": 30440714, 2024-08-20T21:42:56.5429657Z "digest": "sha256:762bedf4b1b784c3de6c5022c5307d63123d3b7cdd59211317e37e9d477deaa0" 2024-08-20T21:42:56.5430285Z }, 2024-08-20T21:42:56.5430526Z { 2024-08-20T21:42:56.5430965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5431535Z "size": 1822, 2024-08-20T21:42:56.5432092Z "digest": "sha256:347beb78d39bc9079a3c20e891908ad5695cdea4a312b0e1de586110bb9ee72f" 2024-08-20T21:42:56.5432729Z }, 2024-08-20T21:42:56.5432971Z { 2024-08-20T21:42:56.5433416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5433969Z "size": 313164873, 2024-08-20T21:42:56.5434539Z "digest": "sha256:71b614615bcb09bc43ad35a0238b1b82945122c70d95efa9728a6a044a7ddb73" 2024-08-20T21:42:56.5435172Z }, 2024-08-20T21:42:56.5435410Z { 2024-08-20T21:42:56.5435825Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5436399Z "size": 803, 2024-08-20T21:42:56.5436951Z "digest": "sha256:f96c8558662d14e2cc064f94a0166dbf827cf4b783c8d3edb4dbc19dcfff8ca3" 2024-08-20T21:42:56.5437575Z }, 2024-08-20T21:42:56.5437814Z { 2024-08-20T21:42:56.5438248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5438825Z "size": 106, 2024-08-20T21:42:56.5439360Z "digest": "sha256:c81ff9d20d36fbd48a567918c08f020d23bd83515aec7d23fda14cd56786be81" 2024-08-20T21:42:56.5440002Z }, 2024-08-20T21:42:56.5440240Z { 2024-08-20T21:42:56.5440842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5441605Z "size": 544, 2024-08-20T21:42:56.5442151Z "digest": "sha256:151e9a9d289467dccd8eefb461af7188b39953d6ac10b678133551bd20586ce6" 2024-08-20T21:42:56.5442789Z }, 2024-08-20T21:42:56.5443013Z { 2024-08-20T21:42:56.5443447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5444021Z "size": 1214, 2024-08-20T21:42:56.5444595Z "digest": "sha256:8538dcd89f0cac92de23cde0e22b0e7cfde631fbef3062f6387bbc78c4b8c051" 2024-08-20T21:42:56.5445291Z }, 2024-08-20T21:42:56.5445534Z { 2024-08-20T21:42:56.5445975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5446534Z "size": 483, 2024-08-20T21:42:56.5447069Z "digest": "sha256:cd13007f6f0922ba40ddf0f45d8693c116a5f27e894bd04a34e40eb57bd5c640" 2024-08-20T21:42:56.5447703Z }, 2024-08-20T21:42:56.5448237Z + exit 0 2024-08-20T21:42:56.5448488Z { 2024-08-20T21:42:56.5448912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5449482Z "size": 110321925, 2024-08-20T21:42:56.5450058Z "digest": "sha256:74fd96b6ebd997dccc9767b9222e3138502d240bca371c2dab11ed193140d9cb" 2024-08-20T21:42:56.5450694Z }, 2024-08-20T21:42:56.5450914Z { 2024-08-20T21:42:56.5451349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5451912Z "size": 3840, 2024-08-20T21:42:56.5452466Z "digest": "sha256:ce4e1800db98e945fe1b38aac9ac5afcc6edcf1c9cc321499bcb108d04fe1442" 2024-08-20T21:42:56.5453117Z }, 2024-08-20T21:42:56.5453351Z { 2024-08-20T21:42:56.5453786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5454599Z "size": 1910, 2024-08-20T21:42:56.5455137Z "digest": "sha256:402b067f30bf9e459448062a2a55e42169b09275d1fa427073a332adc6929112" 2024-08-20T21:42:56.5455758Z }, 2024-08-20T21:42:56.5456000Z { 2024-08-20T21:42:56.5456419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5456995Z "size": 701, 2024-08-20T21:42:56.5457559Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.5458194Z }, 2024-08-20T21:42:56.5458431Z { 2024-08-20T21:42:56.5458972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5459705Z "size": 2663407704, 2024-08-20T21:42:56.5460278Z "digest": "sha256:c6e13ff8aa9a65d1d347f8a336df5ca7febab9d34a3027c2b3c138c1a711baf4" 2024-08-20T21:42:56.5460929Z }, 2024-08-20T21:42:56.5461166Z { 2024-08-20T21:42:56.5461609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5462154Z "size": 32, 2024-08-20T21:42:56.5462707Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5463347Z }, 2024-08-20T21:42:56.5463583Z { 2024-08-20T21:42:56.5464000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5464562Z "size": 379, 2024-08-20T21:42:56.5465136Z "digest": "sha256:e238df05ddcc0ec1e1ce769b25f7103f76af6aa35e4aa4ccd1adcdb7621158f6" 2024-08-20T21:42:56.5465768Z }, 2024-08-20T21:42:56.5466004Z { 2024-08-20T21:42:56.5466437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5467002Z "size": 222887, 2024-08-20T21:42:56.5467541Z "digest": "sha256:a88675bdc72381c487323d232044cca10dd1f77e62eaf00a650bd28d27d6f9bf" 2024-08-20T21:42:56.5468167Z }, 2024-08-20T21:42:56.5468405Z { 2024-08-20T21:42:56.5468839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5469387Z "size": 231, 2024-08-20T21:42:56.5469926Z "digest": "sha256:f4cb80c411d1b5fb90688374327599a7de6ea89c800473afb1b9766003d5a39a" 2024-08-20T21:42:56.5470551Z }, 2024-08-20T21:42:56.5470892Z { 2024-08-20T21:42:56.5471315Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5471865Z "size": 2933275, 2024-08-20T21:42:56.5472539Z "digest": "sha256:c9589ebd9ae736462b4dc9ea4763b2892eae668c19ca321e7c9ef2762b86a528" 2024-08-20T21:42:56.5473149Z }, 2024-08-20T21:42:56.5473383Z { 2024-08-20T21:42:56.5473807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5474360Z "size": 1805, 2024-08-20T21:42:56.5474881Z "digest": "sha256:dd51c681db1409845c172ac4fc813dfc9db27bc797fe919a47724410acd481c4" 2024-08-20T21:42:56.5475499Z }, 2024-08-20T21:42:56.5475735Z { 2024-08-20T21:42:56.5476143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5476872Z "size": 105, 2024-08-20T21:42:56.5477416Z "digest": "sha256:fc983e33a069ae2891a2161f9d24540f9e2728dddc9fd0e63c5b82454ab51f61" 2024-08-20T21:42:56.5478040Z }, 2024-08-20T21:42:56.5478263Z { 2024-08-20T21:42:56.5478694Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5479256Z "size": 564, 2024-08-20T21:42:56.5479802Z "digest": "sha256:bc705d688c776732ddd22c3540b0e135d7dc23e4d46b64918be11b2fa1a265c5" 2024-08-20T21:42:56.5480415Z }, 2024-08-20T21:42:56.5480649Z { 2024-08-20T21:42:56.5481083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5481645Z "size": 32, 2024-08-20T21:42:56.5482177Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5482818Z }, 2024-08-20T21:42:56.5483055Z { 2024-08-20T21:42:56.5483475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5484037Z "size": 104, 2024-08-20T21:42:56.5484574Z "digest": "sha256:99f6c1ac758215178233da5104ddb298caf3ae441129209a625e9e649463b980" 2024-08-20T21:42:56.5485190Z }, 2024-08-20T21:42:56.5485409Z { 2024-08-20T21:42:56.5485841Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5486406Z "size": 504, 2024-08-20T21:42:56.5486962Z "digest": "sha256:c409d27a5a9560bf329dec2ed58b701a43f1ffa8babc7ca13c61321cf22ef007" 2024-08-20T21:42:56.5487589Z }, 2024-08-20T21:42:56.5487832Z { 2024-08-20T21:42:56.5488266Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5488830Z "size": 101109909, 2024-08-20T21:42:56.5489397Z "digest": "sha256:a3a398ca1120cf9319dbbb81b12ddc41f7bf3b5bd3f3612c17c2bf7b5194e505" 2024-08-20T21:42:56.5490036Z }, 2024-08-20T21:42:56.5490368Z { 2024-08-20T21:42:56.5490787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5491349Z "size": 109, 2024-08-20T21:42:56.5491892Z "digest": "sha256:314875f5fae452a321bd50b8527ebb8cf7db0a53ff30293472014390479fcf64" 2024-08-20T21:42:56.5492522Z }, 2024-08-20T21:42:56.5492742Z { 2024-08-20T21:42:56.5493176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5493743Z "size": 490, 2024-08-20T21:42:56.5494295Z "digest": "sha256:8a5e90dafefa97ef114ac2d4c2f8e000536425318010f40d15bfdc6f8d7e2a79" 2024-08-20T21:42:56.5494913Z }, 2024-08-20T21:42:56.5495148Z { 2024-08-20T21:42:56.5495589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5496139Z "size": 333, 2024-08-20T21:42:56.5496694Z "digest": "sha256:e3b946ab3378ddc5e96c6355bd0aaa135f29b1ea34b417652442c9df42b6eb49" 2024-08-20T21:42:56.5497331Z }, 2024-08-20T21:42:56.5497567Z { 2024-08-20T21:42:56.5497989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5498551Z "size": 103, 2024-08-20T21:42:56.5499190Z "digest": "sha256:14a85a990f7189af056863e625917e6a5583ce8a7b1d319783011803ea1a9da4" 2024-08-20T21:42:56.5499852Z }, 2024-08-20T21:42:56.5500072Z { 2024-08-20T21:42:56.5500503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5501066Z "size": 1473, 2024-08-20T21:42:56.5501606Z "digest": "sha256:16d809b167d49638350b86af396a05eca356ba903ca4953a083d29f28154c16a" 2024-08-20T21:42:56.5502209Z }, 2024-08-20T21:42:56.5502446Z { 2024-08-20T21:42:56.5502970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5503520Z "size": 611525039, 2024-08-20T21:42:56.5504092Z "digest": "sha256:9b1a95eedb8338e08a9ba00b933ef64420d41664b1a4c2a40f1703c79a3d2ba1" 2024-08-20T21:42:56.5504726Z }, 2024-08-20T21:42:56.5504967Z { 2024-08-20T21:42:56.5505384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5505951Z "size": 164, 2024-08-20T21:42:56.5506485Z "digest": "sha256:b35ca05427236370ed98f905cf9b8a0e4599173ef1843504fc2547e9525cd483" 2024-08-20T21:42:56.5507106Z }, 2024-08-20T21:42:56.5507327Z { 2024-08-20T21:42:56.5507758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5508320Z "size": 2210, 2024-08-20T21:42:56.5508866Z "digest": "sha256:bad10ac3a6279679db06f845b321a9fdaef80be969acbfd257f9e807762953a8" 2024-08-20T21:42:56.5509506Z }, 2024-08-20T21:42:56.5509742Z { 2024-08-20T21:42:56.5510176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5510726Z "size": 152, 2024-08-20T21:42:56.5511280Z "digest": "sha256:943da2164c7a7bf690bcbc751afe5ea646925ddf0ab96fd2ce48b0a52bd66e9d" 2024-08-20T21:42:56.5512036Z }, 2024-08-20T21:42:56.5512268Z { 2024-08-20T21:42:56.5512675Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5513232Z "size": 648, 2024-08-20T21:42:56.5513767Z "digest": "sha256:2d276441bd8ae528772f4fd876af429ab3518bebf61090795ad3b6aced9d60e4" 2024-08-20T21:42:56.5514379Z }, 2024-08-20T21:42:56.5514595Z { 2024-08-20T21:42:56.5515016Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5515559Z "size": 32, 2024-08-20T21:42:56.5516078Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5516694Z }, 2024-08-20T21:42:56.5516926Z { 2024-08-20T21:42:56.5517346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5517878Z "size": 167, 2024-08-20T21:42:56.5518418Z "digest": "sha256:a3ef8d7d02be75588561825a22ecbdc3129ad8a543f4d1415fee8284d91327bb" 2024-08-20T21:42:56.5519034Z }, 2024-08-20T21:42:56.5519266Z { 2024-08-20T21:42:56.5519673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5520219Z "size": 109, 2024-08-20T21:42:56.5520838Z "digest": "sha256:6aaed7a7874900b7539103d4ef762ab0f9a2d760c898f2ebcf135b73858aeed5" 2024-08-20T21:42:56.5521435Z }, 2024-08-20T21:42:56.5521671Z { 2024-08-20T21:42:56.5522094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5522647Z "size": 102, 2024-08-20T21:42:56.5523256Z "digest": "sha256:944e780839169e63df50b5cc05cc9fb99e8ed0b9692d56cee9cd06137bdb08e3" 2024-08-20T21:42:56.5523839Z }, 2024-08-20T21:42:56.5524063Z { 2024-08-20T21:42:56.5524465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5524970Z "size": 422, 2024-08-20T21:42:56.5525472Z "digest": "sha256:9602de36240c11e5eb9a1926c6443a2ff43948a958a15162e7aa827e1f83ad3e" 2024-08-20T21:42:56.5526045Z }, 2024-08-20T21:42:56.5526266Z { 2024-08-20T21:42:56.5526650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5527175Z "size": 32, 2024-08-20T21:42:56.5527684Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5528263Z }, 2024-08-20T21:42:56.5528487Z { 2024-08-20T21:42:56.5528891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5529416Z "size": 111, 2024-08-20T21:42:56.5529903Z "digest": "sha256:6d3ac49638ec41e1302ba5b1028a55629b0bb2cb2603212aba86d0058783ab50" 2024-08-20T21:42:56.5530484Z }, 2024-08-20T21:42:56.5530705Z { 2024-08-20T21:42:56.5531109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5531617Z "size": 473, 2024-08-20T21:42:56.5532257Z "digest": "sha256:c5c5dbfb1dcc6e7ef2cabd2a583442e21c5b95a0cb625f0766592fff0b807d95" 2024-08-20T21:42:56.5532873Z }, 2024-08-20T21:42:56.5533079Z { 2024-08-20T21:42:56.5533485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5534011Z "size": 32, 2024-08-20T21:42:56.5534526Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5535110Z }, 2024-08-20T21:42:56.5535334Z { 2024-08-20T21:42:56.5535739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5536255Z "size": 112, 2024-08-20T21:42:56.5536754Z "digest": "sha256:327bd4a08d3c4e9ae265be2b8ce0d8ce4e4016f60e4c02a36c638789adf49ea5" 2024-08-20T21:42:56.5537346Z }, 2024-08-20T21:42:56.5537572Z { 2024-08-20T21:42:56.5537958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5538479Z "size": 566, 2024-08-20T21:42:56.5539267Z "digest": "sha256:6c520cbfb1595425e6e07ad7e2952222365e91d02cf25a4ec1f8565279b941de" 2024-08-20T21:42:56.5539983Z }, 2024-08-20T21:42:56.5540204Z { 2024-08-20T21:42:56.5540639Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5541208Z "size": 43163722, 2024-08-20T21:42:56.5541776Z "digest": "sha256:a8cda4c989262a43930b0bf2507986800779f6a6ec65f8ef64bbfa72c5ca5bf6" 2024-08-20T21:42:56.5542398Z }, 2024-08-20T21:42:56.5542634Z { 2024-08-20T21:42:56.5543071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5543633Z "size": 106, 2024-08-20T21:42:56.5544162Z "digest": "sha256:3d995c530b49c1eecd803a2cfde69821f88cc0b8b5429d8955a65319217337d6" 2024-08-20T21:42:56.5544789Z }, 2024-08-20T21:42:56.5545028Z { 2024-08-20T21:42:56.5545444Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5546051Z "size": 295, 2024-08-20T21:42:56.5546581Z "digest": "sha256:51635df36934a1409214ed1d1c03c997d8597887f34e547bca5fb5373e509ca8" 2024-08-20T21:42:56.5547199Z }, 2024-08-20T21:42:56.5547421Z { 2024-08-20T21:42:56.5547856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5548422Z "size": 32, 2024-08-20T21:42:56.5549176Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5549796Z }, 2024-08-20T21:42:56.5550028Z { 2024-08-20T21:42:56.5550545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5551112Z "size": 106, 2024-08-20T21:42:56.5551738Z "digest": "sha256:58f9d093cf6898c8d0fe441806e8afb6fcf8180d6ce0e61f3f8e1a5d27872b95" 2024-08-20T21:42:56.5552330Z }, 2024-08-20T21:42:56.5552553Z { 2024-08-20T21:42:56.5552939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5553456Z "size": 425, 2024-08-20T21:42:56.5553975Z "digest": "sha256:db291b17fdf0ea8442b4ee4ccf970b6bcbbf230840366e5dfe62faf43175568d" 2024-08-20T21:42:56.5554927Z }, 2024-08-20T21:42:56.5555149Z { 2024-08-20T21:42:56.5555713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5556279Z "size": 19308700, 2024-08-20T21:42:56.5556843Z "digest": "sha256:4390932d47dc99aad011fc417cb24b8cd895f84e0247e7b1930a974848f4ae49" 2024-08-20T21:42:56.5557453Z }, 2024-08-20T21:42:56.5557692Z { 2024-08-20T21:42:56.5558121Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5558666Z "size": 108, 2024-08-20T21:42:56.5559202Z "digest": "sha256:d959da1ed4a86d0902fc218b87883ea901725726db56d746621bfeb206d18ad6" 2024-08-20T21:42:56.5559827Z }, 2024-08-20T21:42:56.5560060Z { 2024-08-20T21:42:56.5560477Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5561041Z "size": 438, 2024-08-20T21:42:56.5561571Z "digest": "sha256:82769435ccd5317b20def22a2e4295f283bac4ed179599487713032c3609f189" 2024-08-20T21:42:56.5562186Z }, 2024-08-20T21:42:56.5562407Z { 2024-08-20T21:42:56.5562979Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5563548Z "size": 701, 2024-08-20T21:42:56.5564111Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.5564741Z }, 2024-08-20T21:42:56.5564979Z { 2024-08-20T21:42:56.5565413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5565962Z "size": 141, 2024-08-20T21:42:56.5566502Z "digest": "sha256:50078e7792d3b05b7c50ff90f27eb679dee945052c25d2fa8590cbd90dee9ac0" 2024-08-20T21:42:56.5567239Z }, 2024-08-20T21:42:56.5567459Z { 2024-08-20T21:42:56.5567845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5568361Z "size": 135, 2024-08-20T21:42:56.5568856Z "digest": "sha256:557fee3a8871f019075f948770fb5f0860218079c7dd47fcc7982499991fb2a1" 2024-08-20T21:42:56.5569426Z }, 2024-08-20T21:42:56.5569630Z { 2024-08-20T21:42:56.5570027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5570557Z "size": 4841634229, 2024-08-20T21:42:56.5571081Z "digest": "sha256:cc0b3fc4f447d5036748372379999ebaac2ba414c77302c2910751054210e216" 2024-08-20T21:42:56.5571642Z }, 2024-08-20T21:42:56.5571861Z { 2024-08-20T21:42:56.5572262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5572772Z "size": 194, 2024-08-20T21:42:56.5573267Z "digest": "sha256:f18a0119669a20a0f6987e879f10f84bd03b89b7865214f4904cd1b9987d0b0e" 2024-08-20T21:42:56.5573835Z }, 2024-08-20T21:42:56.5574058Z { 2024-08-20T21:42:56.5574444Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5574962Z "size": 1379, 2024-08-20T21:42:56.5575472Z "digest": "sha256:6197a6a036aca1f62c286b2b81941a497fa5ffed336bca2675084fe69f48f405" 2024-08-20T21:42:56.5576056Z }, 2024-08-20T21:42:56.5576258Z { 2024-08-20T21:42:56.5576656Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5577176Z "size": 701, 2024-08-20T21:42:56.5577685Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.5578284Z }, 2024-08-20T21:42:56.5578505Z { 2024-08-20T21:42:56.5579192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5579739Z "size": 139, 2024-08-20T21:42:56.5580416Z "digest": "sha256:ebdeb237156e5f92ea7a5aa15d85a86b5c64ae0e6ca749609008f2e6eee128b8" 2024-08-20T21:42:56.5581064Z }, 2024-08-20T21:42:56.5581308Z { 2024-08-20T21:42:56.5581721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5582286Z "size": 3733332465, 2024-08-20T21:42:56.5582866Z "digest": "sha256:fd5d0d6ba583ac4b9de591cc4925403acd2588d10fc04114b681beb022597580" 2024-08-20T21:42:56.5583503Z }, 2024-08-20T21:42:56.5583726Z { 2024-08-20T21:42:56.5584157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5584717Z "size": 157, 2024-08-20T21:42:56.5585250Z "digest": "sha256:6b068755e588d5e5994af07fe71bd40adbbc544f6238dea5511e69862a5236dd" 2024-08-20T21:42:56.5585880Z }, 2024-08-20T21:42:56.5586119Z { 2024-08-20T21:42:56.5586551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5587098Z "size": 849, 2024-08-20T21:42:56.5587633Z "digest": "sha256:371497d14c4e377a762eec74e0c7575051758f8aa74cdc2f2436fb382df849d3" 2024-08-20T21:42:56.5588251Z }, 2024-08-20T21:42:56.5588488Z { 2024-08-20T21:42:56.5588902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5589463Z "size": 701, 2024-08-20T21:42:56.5590026Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.5590654Z }, 2024-08-20T21:42:56.5590887Z { 2024-08-20T21:42:56.5591319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5591964Z "size": 139, 2024-08-20T21:42:56.5592443Z "digest": "sha256:4b50cb25a6ac9133e2c7894845496771a2f9b23eb35a58a778bb989f9bfbac5e" 2024-08-20T21:42:56.5593098Z }, 2024-08-20T21:42:56.5593319Z { 2024-08-20T21:42:56.5593724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5594226Z "size": 32, 2024-08-20T21:42:56.5594736Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5595334Z }, 2024-08-20T21:42:56.5595536Z { 2024-08-20T21:42:56.5595937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5596459Z "size": 159, 2024-08-20T21:42:56.5596962Z "digest": "sha256:889d8139c009597aca470a64bf7aaea6d509fd8ee9a96187fa784a1d9931f350" 2024-08-20T21:42:56.5597530Z }, 2024-08-20T21:42:56.5597750Z { 2024-08-20T21:42:56.5598152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5598673Z "size": 907, 2024-08-20T21:42:56.5599175Z "digest": "sha256:2bcabd118f67bf8b2b8c6eb97e245fef2c7753e07e4ecf062b3cf6c905bdea81" 2024-08-20T21:42:56.5599773Z }, 2024-08-20T21:42:56.5600000Z { 2024-08-20T21:42:56.5617741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5618308Z "size": 701, 2024-08-20T21:42:56.5618979Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.5619791Z }, 2024-08-20T21:42:56.5620053Z { 2024-08-20T21:42:56.5620492Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5621053Z "size": 134, 2024-08-20T21:42:56.5621596Z "digest": "sha256:1d831c1ea7c74b810f149c31d6eebc2d2bed6a5fb9179a9b230fec1c85f1094e" 2024-08-20T21:42:56.5622237Z }, 2024-08-20T21:42:56.5622470Z { 2024-08-20T21:42:56.5622890Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5623456Z "size": 32, 2024-08-20T21:42:56.5624002Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5624640Z }, 2024-08-20T21:42:56.5624861Z { 2024-08-20T21:42:56.5625307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5625871Z "size": 158, 2024-08-20T21:42:56.5626405Z "digest": "sha256:d29d428148005386a1f6f19dd2590d9cbf58a6c1f94898e1e6ac92ea666f1641" 2024-08-20T21:42:56.5627011Z }, 2024-08-20T21:42:56.5627244Z { 2024-08-20T21:42:56.5627830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5628373Z "size": 1579, 2024-08-20T21:42:56.5628922Z "digest": "sha256:10df3ed01753e29d9ea7d799510d1d4534671a9a4ac53db74998c1ba9c38a48c" 2024-08-20T21:42:56.5629551Z }, 2024-08-20T21:42:56.5629791Z { 2024-08-20T21:42:56.5630210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5630770Z "size": 32, 2024-08-20T21:42:56.5631436Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5632027Z }, 2024-08-20T21:42:56.5632232Z { 2024-08-20T21:42:56.5632635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5633152Z "size": 136, 2024-08-20T21:42:56.5633643Z "digest": "sha256:3b3541d6658c782e6cf71ee8b9268b351f32109e8571447dc8993e649742d905" 2024-08-20T21:42:56.5634193Z }, 2024-08-20T21:42:56.5634411Z { 2024-08-20T21:42:56.5634812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5635502Z "size": 378, 2024-08-20T21:42:56.5636015Z "digest": "sha256:5e997601a6320f613b759c5684ae83fa39646d3c85a349797faba2e4fbc70d83" 2024-08-20T21:42:56.5636620Z }, 2024-08-20T21:42:56.5636852Z { 2024-08-20T21:42:56.5637255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5637802Z "size": 32, 2024-08-20T21:42:56.5638334Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5638954Z }, 2024-08-20T21:42:56.5639169Z { 2024-08-20T21:42:56.5639589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5640227Z "size": 104, 2024-08-20T21:42:56.5640740Z "digest": "sha256:d2924613594f110f7aa5b1681ba6bc1044cd355b5eeb17944cd11531d33a1552" 2024-08-20T21:42:56.5641351Z }, 2024-08-20T21:42:56.5641588Z { 2024-08-20T21:42:56.5642013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5642551Z "size": 1841, 2024-08-20T21:42:56.5643089Z "digest": "sha256:f66c8294ffd00da25739af287f89a292ee2b06597012e6abac6947e2197961ae" 2024-08-20T21:42:56.5643705Z }, 2024-08-20T21:42:56.5643938Z { 2024-08-20T21:42:56.5644346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5644895Z "size": 7529594, 2024-08-20T21:42:56.5645455Z "digest": "sha256:9da0ec93b0d7d37059844ca82dd8b0ce33bdd51614cc9bd66c391ffe64094ac7" 2024-08-20T21:42:56.5646081Z }, 2024-08-20T21:42:56.5646300Z { 2024-08-20T21:42:56.5646723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5647270Z "size": 106, 2024-08-20T21:42:56.5647779Z "digest": "sha256:d476e4b8627568a99c85e93d60a7ce5676a42e9ee10a5b1054348658d77f56f3" 2024-08-20T21:42:56.5648382Z }, 2024-08-20T21:42:56.5648617Z { 2024-08-20T21:42:56.5649041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5649726Z "size": 163, 2024-08-20T21:42:56.5650258Z "digest": "sha256:397c513e7e07f5c83f0d0c8c407a7bdc3e7167a57744262cd5a1282ee451d52d" 2024-08-20T21:42:56.5650875Z }, 2024-08-20T21:42:56.5651108Z { 2024-08-20T21:42:56.5651512Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5652059Z "size": 7943, 2024-08-20T21:42:56.5652596Z "digest": "sha256:5d6d58e6432b6e8048d3f6cccb787a1cfe9d1203f742a9f0fddfb00338cccae7" 2024-08-20T21:42:56.5653198Z }, 2024-08-20T21:42:56.5653427Z { 2024-08-20T21:42:56.5653845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5654764Z "size": 8066, 2024-08-20T21:42:56.5655315Z "digest": "sha256:f9db7741af1464b9e28a0bab667a6b4559ab7f3d319add7a50bc1dcc2b1e97fe" 2024-08-20T21:42:56.5655952Z }, 2024-08-20T21:42:56.5656185Z { 2024-08-20T21:42:56.5656618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5657166Z "size": 303, 2024-08-20T21:42:56.5657700Z "digest": "sha256:c0f5c95392e3593f2bccab197736341bfef11427975492f581c088b8e21401a1" 2024-08-20T21:42:56.5658485Z }, 2024-08-20T21:42:56.5658799Z { 2024-08-20T21:42:56.5659232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5659795Z "size": 32, 2024-08-20T21:42:56.5660341Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5660959Z }, 2024-08-20T21:42:56.5661197Z { 2024-08-20T21:42:56.5661624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5662181Z "size": 108, 2024-08-20T21:42:56.5662710Z "digest": "sha256:d6d4b368f5e67517827a02d0e324ced915843fa69780cbf87cac205ffcf04032" 2024-08-20T21:42:56.5663333Z }, 2024-08-20T21:42:56.5663566Z { 2024-08-20T21:42:56.5664000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5664546Z "size": 54145664, 2024-08-20T21:42:56.5665110Z "digest": "sha256:6e341d8be4cd61d3439f2becbc390d9d27179704631d29225ba4c01bc518ff56" 2024-08-20T21:42:56.5665740Z }, 2024-08-20T21:42:56.5665957Z { 2024-08-20T21:42:56.5666385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.5666940Z "size": 32, 2024-08-20T21:42:56.5667486Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.5668099Z } 2024-08-20T21:42:56.5668332Z ] 2024-08-20T21:42:56.5668561Z } 2024-08-20T21:42:56.5767492Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:42:56.5767934Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:42:56.5768438Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T21:42:56.5774635Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.5775098Z env: 2024-08-20T21:42:56.5775346Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.5776387Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.5777453Z ##[endgroup] 2024-08-20T21:42:56.5804452Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.5856106Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T21:42:56.5856683Z with: 2024-08-20T21:42:56.5857706Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.5859105Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.5859622Z env: 2024-08-20T21:42:56.5859885Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.5860217Z ##[endgroup] 2024-08-20T21:42:56.5876087Z ##[group]Run set -x 2024-08-20T21:42:56.5876402Z set -x 2024-08-20T21:42:56.5876674Z set +e 2024-08-20T21:42:56.5876938Z  2024-08-20T21:42:56.5877169Z login() { 2024-08-20T21:42:56.5877795Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:42:56.5878473Z } 2024-08-20T21:42:56.5878719Z  2024-08-20T21:42:56.5879005Z retry () { 2024-08-20T21:42:56.5879357Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:42:56.5879748Z } 2024-08-20T21:42:56.5879988Z  2024-08-20T21:42:56.5880266Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:42:56.5880630Z  2024-08-20T21:42:56.5880868Z set -e 2024-08-20T21:42:56.5881296Z # ignore output since only exit code is used for conditional 2024-08-20T21:42:56.5881943Z # only pull docker image if it's not available locally 2024-08-20T21:42:56.5882641Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T21:42:56.5883290Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T21:42:56.5883692Z fi 2024-08-20T21:42:56.5888938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.5889570Z env: 2024-08-20T21:42:56.5889828Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.5890907Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.5892148Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.5892652Z ##[endgroup] 2024-08-20T21:42:56.5915363Z + set +e 2024-08-20T21:42:56.5916314Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.5917005Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.5919442Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:42:56.5920763Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:57.0843081Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:57.0843985Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:57.0844896Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:57.0845431Z 2024-08-20T21:42:57.0845574Z Login Succeeded 2024-08-20T21:42:57.0857850Z + set -e 2024-08-20T21:42:57.0859234Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.0987035Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.0989133Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.3246705Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks 2024-08-20T21:42:57.3253327Z 762bedf4b1b7: Pulling fs layer 2024-08-20T21:42:57.3253925Z 347beb78d39b: Pulling fs layer 2024-08-20T21:42:57.3254701Z 71b614615bcb: Pulling fs layer 2024-08-20T21:42:57.3255688Z f96c8558662d: Pulling fs layer 2024-08-20T21:42:57.3256143Z c81ff9d20d36: Pulling fs layer 2024-08-20T21:42:57.3256763Z 151e9a9d2894: Pulling fs layer 2024-08-20T21:42:57.3257275Z 8538dcd89f0c: Pulling fs layer 2024-08-20T21:42:57.3257753Z cd13007f6f09: Pulling fs layer 2024-08-20T21:42:57.3258131Z 74fd96b6ebd9: Pulling fs layer 2024-08-20T21:42:57.3258509Z ce4e1800db98: Pulling fs layer 2024-08-20T21:42:57.3258962Z 402b067f30bf: Pulling fs layer 2024-08-20T21:42:57.3259439Z 2cebdfe4c5a8: Pulling fs layer 2024-08-20T21:42:57.3259808Z 151e9a9d2894: Waiting 2024-08-20T21:42:57.3260201Z c6e13ff8aa9a: Pulling fs layer 2024-08-20T21:42:57.3260729Z 4f4fb700ef54: Pulling fs layer 2024-08-20T21:42:57.3261340Z 8538dcd89f0c: Waiting 2024-08-20T21:42:57.3261928Z e238df05ddcc: Pulling fs layer 2024-08-20T21:42:57.3262461Z f96c8558662d: Waiting 2024-08-20T21:42:57.3262816Z a88675bdc723: Pulling fs layer 2024-08-20T21:42:57.3263175Z cd13007f6f09: Waiting 2024-08-20T21:42:57.3263493Z 74fd96b6ebd9: Waiting 2024-08-20T21:42:57.3263823Z f4cb80c411d1: Pulling fs layer 2024-08-20T21:42:57.3264247Z 2cebdfe4c5a8: Waiting 2024-08-20T21:42:57.3264597Z c9589ebd9ae7: Pulling fs layer 2024-08-20T21:42:57.3264983Z dd51c681db14: Pulling fs layer 2024-08-20T21:42:57.3265428Z ce4e1800db98: Waiting 2024-08-20T21:42:57.3265748Z fc983e33a069: Pulling fs layer 2024-08-20T21:42:57.3266121Z c6e13ff8aa9a: Waiting 2024-08-20T21:42:57.3266446Z 402b067f30bf: Waiting 2024-08-20T21:42:57.3266767Z bc705d688c77: Pulling fs layer 2024-08-20T21:42:57.3267137Z c81ff9d20d36: Waiting 2024-08-20T21:42:57.3267464Z 4f4fb700ef54: Waiting 2024-08-20T21:42:57.3267787Z e238df05ddcc: Waiting 2024-08-20T21:42:57.3268090Z a88675bdc723: Waiting 2024-08-20T21:42:57.3268425Z 99f6c1ac7582: Pulling fs layer 2024-08-20T21:42:57.3268799Z f4cb80c411d1: Waiting 2024-08-20T21:42:57.3269367Z c409d27a5a95: Pulling fs layer 2024-08-20T21:42:57.3269731Z a3a398ca1120: Pulling fs layer 2024-08-20T21:42:57.3270114Z 314875f5fae4: Pulling fs layer 2024-08-20T21:42:57.3270503Z 8a5e90dafefa: Pulling fs layer 2024-08-20T21:42:57.3270864Z e3b946ab3378: Pulling fs layer 2024-08-20T21:42:57.3271230Z dd51c681db14: Waiting 2024-08-20T21:42:57.3271546Z c9589ebd9ae7: Waiting 2024-08-20T21:42:57.3271880Z 14a85a990f71: Pulling fs layer 2024-08-20T21:42:57.3272240Z 16d809b167d4: Pulling fs layer 2024-08-20T21:42:57.3272615Z 9b1a95eedb83: Pulling fs layer 2024-08-20T21:42:57.3272973Z fc983e33a069: Waiting 2024-08-20T21:42:57.3273281Z b35ca0542723: Pulling fs layer 2024-08-20T21:42:57.3273652Z bad10ac3a627: Pulling fs layer 2024-08-20T21:42:57.3274028Z 943da2164c7a: Pulling fs layer 2024-08-20T21:42:57.3274398Z 2d276441bd8a: Pulling fs layer 2024-08-20T21:42:57.3274752Z a3ef8d7d02be: Pulling fs layer 2024-08-20T21:42:57.3275115Z a3a398ca1120: Waiting 2024-08-20T21:42:57.3275426Z bc705d688c77: Waiting 2024-08-20T21:42:57.3275743Z 314875f5fae4: Waiting 2024-08-20T21:42:57.3276052Z 6aaed7a78749: Pulling fs layer 2024-08-20T21:42:57.3276408Z 99f6c1ac7582: Waiting 2024-08-20T21:42:57.3276724Z 8a5e90dafefa: Waiting 2024-08-20T21:42:57.3277024Z c409d27a5a95: Waiting 2024-08-20T21:42:57.3277347Z 944e78083916: Pulling fs layer 2024-08-20T21:42:57.3277700Z 943da2164c7a: Waiting 2024-08-20T21:42:57.3278013Z e3b946ab3378: Waiting 2024-08-20T21:42:57.3278305Z 2d276441bd8a: Waiting 2024-08-20T21:42:57.3278610Z 14a85a990f71: Waiting 2024-08-20T21:42:57.3278932Z 9602de36240c: Pulling fs layer 2024-08-20T21:42:57.3279288Z 16d809b167d4: Waiting 2024-08-20T21:42:57.3279576Z bad10ac3a627: Waiting 2024-08-20T21:42:57.3279905Z 6d3ac49638ec: Pulling fs layer 2024-08-20T21:42:57.3280268Z a3ef8d7d02be: Waiting 2024-08-20T21:42:57.3280583Z c5c5dbfb1dcc: Pulling fs layer 2024-08-20T21:42:57.3280945Z 6aaed7a78749: Waiting 2024-08-20T21:42:57.3281253Z 944e78083916: Waiting 2024-08-20T21:42:57.3281560Z 9b1a95eedb83: Waiting 2024-08-20T21:42:57.3281872Z 327bd4a08d3c: Pulling fs layer 2024-08-20T21:42:57.3282246Z 6c520cbfb159: Pulling fs layer 2024-08-20T21:42:57.3282602Z 6d3ac49638ec: Waiting 2024-08-20T21:42:57.3282892Z 9602de36240c: Waiting 2024-08-20T21:42:57.3283337Z a8cda4c98926: Pulling fs layer 2024-08-20T21:42:57.3283710Z c5c5dbfb1dcc: Waiting 2024-08-20T21:42:57.3284023Z b35ca0542723: Waiting 2024-08-20T21:42:57.3284318Z 327bd4a08d3c: Waiting 2024-08-20T21:42:57.3284642Z 3d995c530b49: Pulling fs layer 2024-08-20T21:42:57.3285005Z 6c520cbfb159: Waiting 2024-08-20T21:42:57.3285331Z 51635df36934: Pulling fs layer 2024-08-20T21:42:57.3285669Z 3d995c530b49: Waiting 2024-08-20T21:42:57.3285995Z 58f9d093cf68: Pulling fs layer 2024-08-20T21:42:57.3286370Z db291b17fdf0: Pulling fs layer 2024-08-20T21:42:57.3286724Z 4390932d47dc: Pulling fs layer 2024-08-20T21:42:57.3287077Z db291b17fdf0: Waiting 2024-08-20T21:42:57.3287388Z 51635df36934: Waiting 2024-08-20T21:42:57.3287714Z d959da1ed4a8: Pulling fs layer 2024-08-20T21:42:57.3288066Z 82769435ccd5: Pulling fs layer 2024-08-20T21:42:57.3288425Z 58f9d093cf68: Waiting 2024-08-20T21:42:57.3288735Z d959da1ed4a8: Waiting 2024-08-20T21:42:57.3289046Z 4390932d47dc: Waiting 2024-08-20T21:42:57.3289356Z 50078e7792d3: Pulling fs layer 2024-08-20T21:42:57.3289709Z 82769435ccd5: Waiting 2024-08-20T21:42:57.3290170Z 557fee3a8871: Pulling fs layer 2024-08-20T21:42:57.3290782Z cc0b3fc4f447: Pulling fs layer 2024-08-20T21:42:57.3291423Z 50078e7792d3: Waiting 2024-08-20T21:42:57.3291942Z f18a0119669a: Pulling fs layer 2024-08-20T21:42:57.3292556Z cc0b3fc4f447: Waiting 2024-08-20T21:42:57.3293054Z 557fee3a8871: Waiting 2024-08-20T21:42:57.3293619Z 6197a6a036ac: Pulling fs layer 2024-08-20T21:42:57.3294309Z ebdeb237156e: Pulling fs layer 2024-08-20T21:42:57.3294938Z 6197a6a036ac: Waiting 2024-08-20T21:42:57.3295490Z f18a0119669a: Waiting 2024-08-20T21:42:57.3296082Z fd5d0d6ba583: Pulling fs layer 2024-08-20T21:42:57.3296705Z 6b068755e588: Pulling fs layer 2024-08-20T21:42:57.3297307Z 371497d14c4e: Pulling fs layer 2024-08-20T21:42:57.3298088Z ebdeb237156e: Waiting 2024-08-20T21:42:57.3298806Z 4b50cb25a6ac: Pulling fs layer 2024-08-20T21:42:57.3299465Z 6b068755e588: Waiting 2024-08-20T21:42:57.3300053Z 371497d14c4e: Waiting 2024-08-20T21:42:57.3300624Z fd5d0d6ba583: Waiting 2024-08-20T21:42:57.3301164Z 4b50cb25a6ac: Waiting 2024-08-20T21:42:57.3301690Z 889d8139c009: Pulling fs layer 2024-08-20T21:42:57.3302319Z 2bcabd118f67: Pulling fs layer 2024-08-20T21:42:57.3302978Z 2bcabd118f67: Waiting 2024-08-20T21:42:57.3303571Z 1d831c1ea7c7: Pulling fs layer 2024-08-20T21:42:57.3304236Z d29d42814800: Pulling fs layer 2024-08-20T21:42:57.3304886Z 889d8139c009: Waiting 2024-08-20T21:42:57.3305452Z 10df3ed01753: Pulling fs layer 2024-08-20T21:42:57.3306054Z d29d42814800: Waiting 2024-08-20T21:42:57.3306570Z 3b3541d6658c: Pulling fs layer 2024-08-20T21:42:57.3307204Z 1d831c1ea7c7: Waiting 2024-08-20T21:42:57.3307790Z 5e997601a632: Pulling fs layer 2024-08-20T21:42:57.3308410Z 3b3541d6658c: Waiting 2024-08-20T21:42:57.3308966Z 5e997601a632: Waiting 2024-08-20T21:42:57.3309528Z 10df3ed01753: Waiting 2024-08-20T21:42:57.3310068Z d2924613594f: Pulling fs layer 2024-08-20T21:42:57.3310670Z f66c8294ffd0: Pulling fs layer 2024-08-20T21:42:57.3311330Z 9da0ec93b0d7: Pulling fs layer 2024-08-20T21:42:57.3312019Z d476e4b86275: Pulling fs layer 2024-08-20T21:42:57.3312656Z 9da0ec93b0d7: Waiting 2024-08-20T21:42:57.3313250Z 397c513e7e07: Pulling fs layer 2024-08-20T21:42:57.3313927Z 5d6d58e6432b: Pulling fs layer 2024-08-20T21:42:57.3314557Z f9db7741af14: Pulling fs layer 2024-08-20T21:42:57.3315168Z c0f5c95392e3: Pulling fs layer 2024-08-20T21:42:57.3315791Z f66c8294ffd0: Waiting 2024-08-20T21:42:57.3316386Z d6d4b368f5e6: Pulling fs layer 2024-08-20T21:42:57.3317042Z c0f5c95392e3: Waiting 2024-08-20T21:42:57.3317615Z 6e341d8be4cd: Pulling fs layer 2024-08-20T21:42:57.3318273Z f9db7741af14: Waiting 2024-08-20T21:42:57.3318792Z d6d4b368f5e6: Waiting 2024-08-20T21:42:57.3319281Z 6e341d8be4cd: Waiting 2024-08-20T21:42:57.4094715Z 347beb78d39b: Verifying Checksum 2024-08-20T21:42:57.4095421Z 347beb78d39b: Download complete 2024-08-20T21:42:57.6407166Z f96c8558662d: Download complete 2024-08-20T21:42:57.6817595Z 762bedf4b1b7: Download complete 2024-08-20T21:42:57.7481494Z 151e9a9d2894: Verifying Checksum 2024-08-20T21:42:57.7482071Z 151e9a9d2894: Download complete 2024-08-20T21:42:57.7494062Z c81ff9d20d36: Verifying Checksum 2024-08-20T21:42:57.7494647Z c81ff9d20d36: Download complete 2024-08-20T21:42:57.8213112Z 8538dcd89f0c: Verifying Checksum 2024-08-20T21:42:57.8213954Z 8538dcd89f0c: Download complete 2024-08-20T21:42:57.8260562Z cd13007f6f09: Verifying Checksum 2024-08-20T21:42:57.8261242Z cd13007f6f09: Download complete 2024-08-20T21:42:57.8827850Z ce4e1800db98: Download complete 2024-08-20T21:42:57.9489575Z 402b067f30bf: Verifying Checksum 2024-08-20T21:42:57.9490071Z 402b067f30bf: Download complete 2024-08-20T21:42:58.0217300Z 2cebdfe4c5a8: Verifying Checksum 2024-08-20T21:42:58.0217790Z 2cebdfe4c5a8: Download complete 2024-08-20T21:42:58.4015618Z 762bedf4b1b7: Pull complete 2024-08-20T21:42:58.4124676Z 347beb78d39b: Pull complete 2024-08-20T21:42:58.9701183Z 74fd96b6ebd9: Download complete 2024-08-20T21:42:58.9770544Z 4f4fb700ef54: Verifying Checksum 2024-08-20T21:42:58.9771016Z 4f4fb700ef54: Download complete 2024-08-20T21:42:59.0491305Z e238df05ddcc: Verifying Checksum 2024-08-20T21:42:59.0491759Z e238df05ddcc: Download complete 2024-08-20T21:42:59.1298967Z a88675bdc723: Download complete 2024-08-20T21:42:59.1959809Z f4cb80c411d1: Verifying Checksum 2024-08-20T21:42:59.1960243Z f4cb80c411d1: Download complete 2024-08-20T21:42:59.2933836Z c9589ebd9ae7: Verifying Checksum 2024-08-20T21:42:59.2934290Z c9589ebd9ae7: Download complete 2024-08-20T21:42:59.3555682Z dd51c681db14: Download complete 2024-08-20T21:42:59.4129012Z fc983e33a069: Verifying Checksum 2024-08-20T21:42:59.4129436Z fc983e33a069: Download complete 2024-08-20T21:42:59.4926027Z bc705d688c77: Download complete 2024-08-20T21:42:59.5642709Z 99f6c1ac7582: Download complete 2024-08-20T21:42:59.6400861Z c409d27a5a95: Download complete 2024-08-20T21:43:00.5033252Z 71b614615bcb: Verifying Checksum 2024-08-20T21:43:00.5033729Z 71b614615bcb: Download complete 2024-08-20T21:43:00.5730835Z 314875f5fae4: Download complete 2024-08-20T21:43:00.6347813Z 8a5e90dafefa: Verifying Checksum 2024-08-20T21:43:00.6348312Z 8a5e90dafefa: Download complete 2024-08-20T21:43:00.6918159Z a3a398ca1120: Verifying Checksum 2024-08-20T21:43:00.6918695Z a3a398ca1120: Download complete 2024-08-20T21:43:00.7029570Z e3b946ab3378: Verifying Checksum 2024-08-20T21:43:00.7558413Z 14a85a990f71: Download complete 2024-08-20T21:43:00.7844551Z 16d809b167d4: Verifying Checksum 2024-08-20T21:43:00.7845032Z 16d809b167d4: Download complete 2024-08-20T21:43:00.8615930Z b35ca0542723: Verifying Checksum 2024-08-20T21:43:00.8616422Z b35ca0542723: Download complete 2024-08-20T21:43:00.9227260Z bad10ac3a627: Download complete 2024-08-20T21:43:00.9837068Z 943da2164c7a: Download complete 2024-08-20T21:43:01.0595461Z 2d276441bd8a: Download complete 2024-08-20T21:43:01.1217987Z a3ef8d7d02be: Verifying Checksum 2024-08-20T21:43:01.1218463Z a3ef8d7d02be: Download complete 2024-08-20T21:43:01.1956472Z 6aaed7a78749: Verifying Checksum 2024-08-20T21:43:01.1956992Z 6aaed7a78749: Download complete 2024-08-20T21:43:01.2644384Z 944e78083916: Verifying Checksum 2024-08-20T21:43:01.2644839Z 944e78083916: Download complete 2024-08-20T21:43:01.3521677Z 9602de36240c: Verifying Checksum 2024-08-20T21:43:01.3522198Z 9602de36240c: Download complete 2024-08-20T21:43:01.4333947Z 6d3ac49638ec: Verifying Checksum 2024-08-20T21:43:01.4334412Z 6d3ac49638ec: Download complete 2024-08-20T21:43:01.5082564Z c5c5dbfb1dcc: Verifying Checksum 2024-08-20T21:43:01.5083070Z c5c5dbfb1dcc: Download complete 2024-08-20T21:43:01.5758286Z 327bd4a08d3c: Verifying Checksum 2024-08-20T21:43:01.5758760Z 327bd4a08d3c: Download complete 2024-08-20T21:43:01.6616179Z 6c520cbfb159: Download complete 2024-08-20T21:43:02.1438628Z a8cda4c98926: Verifying Checksum 2024-08-20T21:43:02.1439122Z a8cda4c98926: Download complete 2024-08-20T21:43:02.2223315Z 3d995c530b49: Verifying Checksum 2024-08-20T21:43:02.2223821Z 3d995c530b49: Download complete 2024-08-20T21:43:02.2946092Z 51635df36934: Download complete 2024-08-20T21:43:02.3718040Z 58f9d093cf68: Verifying Checksum 2024-08-20T21:43:02.3718580Z 58f9d093cf68: Download complete 2024-08-20T21:43:02.4361353Z db291b17fdf0: Verifying Checksum 2024-08-20T21:43:02.4361841Z db291b17fdf0: Download complete 2024-08-20T21:43:02.6837798Z 4390932d47dc: Verifying Checksum 2024-08-20T21:43:02.6838465Z 4390932d47dc: Download complete 2024-08-20T21:43:02.7523626Z d959da1ed4a8: Download complete 2024-08-20T21:43:02.8218871Z 82769435ccd5: Download complete 2024-08-20T21:43:02.9012241Z 50078e7792d3: Verifying Checksum 2024-08-20T21:43:02.9012925Z 50078e7792d3: Download complete 2024-08-20T21:43:02.9720998Z 557fee3a8871: Download complete 2024-08-20T21:43:06.9122180Z 9b1a95eedb83: Verifying Checksum 2024-08-20T21:43:06.9123339Z 9b1a95eedb83: Download complete 2024-08-20T21:43:06.9898385Z f18a0119669a: Verifying Checksum 2024-08-20T21:43:06.9898924Z f18a0119669a: Download complete 2024-08-20T21:43:07.0560875Z 6197a6a036ac: Verifying Checksum 2024-08-20T21:43:07.0561346Z 6197a6a036ac: Download complete 2024-08-20T21:43:07.1186447Z ebdeb237156e: Verifying Checksum 2024-08-20T21:43:07.1186900Z ebdeb237156e: Download complete 2024-08-20T21:43:07.3684424Z 71b614615bcb: Pull complete 2024-08-20T21:43:07.8015269Z f96c8558662d: Pull complete 2024-08-20T21:43:08.2419964Z c81ff9d20d36: Pull complete 2024-08-20T21:43:08.7303216Z 151e9a9d2894: Pull complete 2024-08-20T21:43:09.0812694Z 8538dcd89f0c: Pull complete 2024-08-20T21:43:09.3372689Z cd13007f6f09: Pull complete 2024-08-20T21:43:11.7061395Z 74fd96b6ebd9: Pull complete 2024-08-20T21:43:12.1028186Z ce4e1800db98: Pull complete 2024-08-20T21:43:12.5106849Z 402b067f30bf: Pull complete 2024-08-20T21:43:12.9709275Z 2cebdfe4c5a8: Pull complete 2024-08-20T21:43:24.7092835Z c6e13ff8aa9a: Verifying Checksum 2024-08-20T21:43:24.7093343Z c6e13ff8aa9a: Download complete 2024-08-20T21:43:24.7930078Z 6b068755e588: Download complete 2024-08-20T21:43:24.8636400Z 371497d14c4e: Verifying Checksum 2024-08-20T21:43:24.8636843Z 371497d14c4e: Download complete 2024-08-20T21:43:24.9386127Z 4b50cb25a6ac: Verifying Checksum 2024-08-20T21:43:24.9386602Z 4b50cb25a6ac: Download complete 2024-08-20T21:43:25.0122413Z 889d8139c009: Download complete 2024-08-20T21:43:25.1164169Z 2bcabd118f67: Verifying Checksum 2024-08-20T21:43:25.1164650Z 2bcabd118f67: Download complete 2024-08-20T21:43:25.1942323Z 1d831c1ea7c7: Verifying Checksum 2024-08-20T21:43:25.1942789Z 1d831c1ea7c7: Download complete 2024-08-20T21:43:25.2581224Z d29d42814800: Verifying Checksum 2024-08-20T21:43:25.3377433Z 10df3ed01753: Verifying Checksum 2024-08-20T21:43:25.3377841Z 10df3ed01753: Download complete 2024-08-20T21:43:25.4127564Z 3b3541d6658c: Verifying Checksum 2024-08-20T21:43:25.4128028Z 3b3541d6658c: Download complete 2024-08-20T21:43:25.5141576Z 5e997601a632: Download complete 2024-08-20T21:43:25.5867128Z d2924613594f: Verifying Checksum 2024-08-20T21:43:25.5867596Z d2924613594f: Download complete 2024-08-20T21:43:25.6477523Z f66c8294ffd0: Verifying Checksum 2024-08-20T21:43:25.6478031Z f66c8294ffd0: Download complete 2024-08-20T21:43:25.7773557Z 9da0ec93b0d7: Verifying Checksum 2024-08-20T21:43:25.7774048Z 9da0ec93b0d7: Download complete 2024-08-20T21:43:25.8656641Z d476e4b86275: Download complete 2024-08-20T21:43:25.9373267Z 397c513e7e07: Verifying Checksum 2024-08-20T21:43:25.9373766Z 397c513e7e07: Download complete 2024-08-20T21:43:26.0119603Z 5d6d58e6432b: Verifying Checksum 2024-08-20T21:43:26.0120091Z 5d6d58e6432b: Download complete 2024-08-20T21:43:26.0900835Z f9db7741af14: Verifying Checksum 2024-08-20T21:43:26.0901334Z f9db7741af14: Download complete 2024-08-20T21:43:26.1527733Z c0f5c95392e3: Verifying Checksum 2024-08-20T21:43:26.1528227Z c0f5c95392e3: Download complete 2024-08-20T21:43:26.2280692Z d6d4b368f5e6: Verifying Checksum 2024-08-20T21:43:26.2281211Z d6d4b368f5e6: Download complete 2024-08-20T21:43:26.8196962Z 6e341d8be4cd: Verifying Checksum 2024-08-20T21:43:26.8197455Z 6e341d8be4cd: Download complete 2024-08-20T21:43:44.5199667Z fd5d0d6ba583: Verifying Checksum 2024-08-20T21:43:44.5200399Z fd5d0d6ba583: Download complete 2024-08-20T21:43:51.4375454Z cc0b3fc4f447: Verifying Checksum 2024-08-20T21:43:51.4375942Z cc0b3fc4f447: Download complete 2024-08-20T21:44:03.2457481Z c6e13ff8aa9a: Pull complete 2024-08-20T21:44:03.7293659Z 4f4fb700ef54: Pull complete 2024-08-20T21:44:04.1756802Z e238df05ddcc: Pull complete 2024-08-20T21:44:04.6483875Z a88675bdc723: Pull complete 2024-08-20T21:44:05.1191461Z f4cb80c411d1: Pull complete 2024-08-20T21:44:05.6248479Z c9589ebd9ae7: Pull complete 2024-08-20T21:44:06.0428484Z dd51c681db14: Pull complete 2024-08-20T21:44:06.5067334Z fc983e33a069: Pull complete 2024-08-20T21:44:06.9688552Z bc705d688c77: Pull complete 2024-08-20T21:44:07.6501948Z 99f6c1ac7582: Pull complete 2024-08-20T21:44:07.8857626Z c409d27a5a95: Pull complete 2024-08-20T21:44:09.8960671Z a3a398ca1120: Pull complete 2024-08-20T21:44:10.2929564Z 314875f5fae4: Pull complete 2024-08-20T21:44:10.6117752Z 8a5e90dafefa: Pull complete 2024-08-20T21:44:10.8788778Z e3b946ab3378: Pull complete 2024-08-20T21:44:11.0822008Z 14a85a990f71: Pull complete 2024-08-20T21:44:11.3266814Z 16d809b167d4: Pull complete 2024-08-20T21:44:18.9589665Z 9b1a95eedb83: Pull complete 2024-08-20T21:44:19.4038698Z b35ca0542723: Pull complete 2024-08-20T21:44:19.8405855Z bad10ac3a627: Pull complete 2024-08-20T21:44:20.2048739Z 943da2164c7a: Pull complete 2024-08-20T21:44:20.5555683Z 2d276441bd8a: Pull complete 2024-08-20T21:44:21.2572883Z a3ef8d7d02be: Pull complete 2024-08-20T21:44:21.4806879Z 6aaed7a78749: Pull complete 2024-08-20T21:44:21.8173413Z 944e78083916: Pull complete 2024-08-20T21:44:22.1455169Z 9602de36240c: Pull complete 2024-08-20T21:44:22.8117678Z 6d3ac49638ec: Pull complete 2024-08-20T21:44:23.2829467Z c5c5dbfb1dcc: Pull complete 2024-08-20T21:44:23.8362419Z 327bd4a08d3c: Pull complete 2024-08-20T21:44:24.2698997Z 6c520cbfb159: Pull complete 2024-08-20T21:44:25.7541179Z a8cda4c98926: Pull complete 2024-08-20T21:44:25.7900845Z 3d995c530b49: Pull complete 2024-08-20T21:44:26.0549347Z 51635df36934: Pull complete 2024-08-20T21:44:26.7064090Z 58f9d093cf68: Pull complete 2024-08-20T21:44:27.1039716Z db291b17fdf0: Pull complete 2024-08-20T21:44:27.7059117Z 4390932d47dc: Pull complete 2024-08-20T21:44:28.0122122Z d959da1ed4a8: Pull complete 2024-08-20T21:44:28.3754707Z 82769435ccd5: Pull complete 2024-08-20T21:44:28.7816682Z 50078e7792d3: Pull complete 2024-08-20T21:44:29.0862191Z 557fee3a8871: Pull complete 2024-08-20T21:44:57.9706051Z cc0b3fc4f447: Pull complete 2024-08-20T21:44:58.2183569Z f18a0119669a: Pull complete 2024-08-20T21:44:58.3692101Z 6197a6a036ac: Pull complete 2024-08-20T21:44:59.2602065Z ebdeb237156e: Pull complete 2024-08-20T21:45:46.4725969Z fd5d0d6ba583: Pull complete 2024-08-20T21:45:46.9597379Z 6b068755e588: Pull complete 2024-08-20T21:45:47.4652507Z 371497d14c4e: Pull complete 2024-08-20T21:45:48.2619082Z 4b50cb25a6ac: Pull complete 2024-08-20T21:45:49.1936871Z 889d8139c009: Pull complete 2024-08-20T21:45:49.6663602Z 2bcabd118f67: Pull complete 2024-08-20T21:45:50.3394679Z 1d831c1ea7c7: Pull complete 2024-08-20T21:45:51.0836135Z d29d42814800: Pull complete 2024-08-20T21:45:51.5053501Z 10df3ed01753: Pull complete 2024-08-20T21:45:52.2738372Z 3b3541d6658c: Pull complete 2024-08-20T21:45:52.7216475Z 5e997601a632: Pull complete 2024-08-20T21:45:53.5241410Z d2924613594f: Pull complete 2024-08-20T21:45:53.8546971Z f66c8294ffd0: Pull complete 2024-08-20T21:45:54.4217656Z 9da0ec93b0d7: Pull complete 2024-08-20T21:45:54.8652912Z d476e4b86275: Pull complete 2024-08-20T21:45:55.2805109Z 397c513e7e07: Pull complete 2024-08-20T21:45:55.6857456Z 5d6d58e6432b: Pull complete 2024-08-20T21:45:56.0205258Z f9db7741af14: Pull complete 2024-08-20T21:45:56.4786633Z c0f5c95392e3: Pull complete 2024-08-20T21:45:57.2838102Z d6d4b368f5e6: Pull complete 2024-08-20T21:45:58.9570395Z 6e341d8be4cd: Pull complete 2024-08-20T21:45:59.6181716Z Digest: sha256:ed18f711327455e269271a17448384275684348a472ba289f00904bd14501fa9 2024-08-20T21:45:59.7067388Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:45:59.7445488Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:45:59.7535776Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:45:59.7536715Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:45:59.7545623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:45:59.7546120Z env: 2024-08-20T21:45:59.7546403Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:59.7546759Z ##[endgroup] 2024-08-20T21:45:59.7804300Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:45:59.7805095Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:45:59.7805806Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T21:45:59.7806476Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:45:59.7812556Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:45:59.7813010Z env: 2024-08-20T21:45:59.7813269Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:59.7813595Z ##[endgroup] 2024-08-20T21:46:00.4288316Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:46:00.6009990Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T21:46:00.6014821Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T21:46:00.7315990Z Prepare all required actions 2024-08-20T21:46:00.7316490Z Getting action download info 2024-08-20T21:46:00.8755486Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T21:46:01.0551496Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T21:46:01.1882452Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T21:46:01.1882904Z with: 2024-08-20T21:46:01.1883188Z name: linux-jammy-py3.8-gcc11-build 2024-08-20T21:46:01.1883589Z s3-bucket: gha-artifacts 2024-08-20T21:46:01.1883900Z env: 2024-08-20T21:46:01.1884134Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:01.1884450Z ##[endgroup] 2024-08-20T21:46:01.2076966Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:46:01.2077363Z with: 2024-08-20T21:46:01.2077633Z name: linux-jammy-py3.8-gcc11-build 2024-08-20T21:46:01.2078012Z s3-bucket: gha-artifacts 2024-08-20T21:46:01.2078362Z region: us-east-1 2024-08-20T21:46:01.2078611Z env: 2024-08-20T21:46:01.2078852Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:01.2079173Z ##[endgroup] 2024-08-20T21:46:01.6686765Z (node:118017) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:46:01.6687587Z 2024-08-20T21:46:01.6687836Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:46:01.6688562Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:46:01.6689464Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:46:01.7344533Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-jammy-py3.8-gcc11-build/ 2024-08-20T21:46:01.7345606Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:46:05.4780091Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:46:05.4785734Z Artifact download has finished successfully 2024-08-20T21:46:05.5076618Z ##[group]Run unzip -o artifacts.zip 2024-08-20T21:46:05.5077051Z unzip -o artifacts.zip 2024-08-20T21:46:05.5082821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:05.5083311Z env: 2024-08-20T21:46:05.5083558Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:05.5083894Z ##[endgroup] 2024-08-20T21:46:05.5299563Z Archive: artifacts.zip 2024-08-20T21:46:05.5300009Z creating: dist/ 2024-08-20T21:46:06.4566545Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-20T21:46:06.4567215Z creating: build/custom_test_artifacts/ 2024-08-20T21:46:06.4567798Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T21:46:06.4568529Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T21:46:06.4569389Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:06.4570419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:06.4571369Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:06.4572366Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:06.4573400Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:06.4574416Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:06.4575565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:06.4576735Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:06.4577777Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:06.4578948Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:06.4580417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:06.4581764Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:06.4582976Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:06.4584176Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:06.4585393Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:06.4586639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:06.4587657Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:06.4588616Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:06.4589579Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T21:46:06.4590658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T21:46:06.4591869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T21:46:06.4593022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T21:46:06.4594107Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T21:46:06.4617274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T21:46:06.4618575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T21:46:06.4619734Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T21:46:06.4620872Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T21:46:06.4621997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T21:46:06.4623111Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T21:46:06.4736469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T21:46:06.4737560Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T21:46:06.4738824Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T21:46:06.4740132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T21:46:06.4741366Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T21:46:06.4742525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T21:46:06.4743716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T21:46:06.4744932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T21:46:06.4746139Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T21:46:06.4747337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T21:46:06.4748519Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T21:46:06.4758185Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T21:46:06.4831770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T21:46:06.4833124Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:06.4834375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:06.4835410Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T21:46:06.4836360Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T21:46:06.4837299Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:06.4838196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T21:46:06.4839003Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T21:46:06.4839826Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T21:46:06.4945566Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T21:46:06.5005393Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T21:46:06.5006472Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T21:46:06.5007192Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T21:46:06.5008030Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:06.5009030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:06.5009985Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:06.5010948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:06.5011953Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:06.5012954Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:06.5014105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:06.5015258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:06.5016285Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:06.5017295Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:06.5018613Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:06.5019838Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:06.5021037Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:06.5022225Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:06.5023440Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:06.5024659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:06.5025678Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:06.5026630Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:06.5027597Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T21:46:06.5028709Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T21:46:06.5029954Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T21:46:06.5031150Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T21:46:06.5032439Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T21:46:06.5033704Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T21:46:06.5034870Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T21:46:06.5036043Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T21:46:06.5037212Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T21:46:06.5038356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T21:46:06.5044278Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T21:46:06.5103750Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T21:46:06.5105028Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:06.5106168Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:06.5107178Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T21:46:06.5108099Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T21:46:06.5109017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:06.5109903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T21:46:06.5110694Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T21:46:06.5111503Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T21:46:06.5156427Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T21:46:06.5157251Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T21:46:06.5158048Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T21:46:06.5158973Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:06.5160040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:06.5161075Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:06.5162122Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:06.5163228Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:06.5164301Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:06.5165524Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:06.5166765Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:06.5167872Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:06.5168965Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:06.5170250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:06.5171547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:06.5172810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:06.5174083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:06.5175540Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:06.5176920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:06.5178020Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:06.5179182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:06.5180246Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T21:46:06.5181442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T21:46:06.5182791Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T21:46:06.5184077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T21:46:06.5185287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T21:46:06.5186541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T21:46:06.5187780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T21:46:06.5189043Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T21:46:06.5190300Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T21:46:06.5191550Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T21:46:06.5192873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T21:46:06.5297363Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T21:46:06.5298735Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T21:46:06.5300036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T21:46:06.5301437Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T21:46:06.5302782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T21:46:06.5304031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T21:46:06.5305348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T21:46:06.5306680Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T21:46:06.5308002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T21:46:06.5309325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T21:46:06.5310617Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T21:46:06.5319298Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T21:46:06.5374829Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T21:46:06.5376253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:06.5377476Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:06.5378829Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T21:46:06.5379936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T21:46:06.5380953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:06.5381922Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T21:46:06.5382790Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T21:46:06.5383674Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T21:46:06.5479215Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T21:46:06.5518922Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T21:46:06.5519594Z creating: build/lib/ 2024-08-20T21:46:06.5519942Z inflating: build/lib/libclog.a 2024-08-20T21:46:06.5527579Z inflating: build/lib/libpthreadpool.a 2024-08-20T21:46:06.5535229Z inflating: build/lib/libcpuinfo.a 2024-08-20T21:46:06.5542851Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T21:46:06.5607116Z inflating: build/lib/libgtest.a 2024-08-20T21:46:06.5612543Z inflating: build/lib/libittnotify.a 2024-08-20T21:46:06.5690328Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T21:46:06.5758951Z inflating: build/lib/libbenchmark.a 2024-08-20T21:46:06.5785110Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T21:46:06.5845521Z inflating: build/lib/libasmjit.a 2024-08-20T21:46:06.5952875Z inflating: build/lib/libgloo.a 2024-08-20T21:46:06.6368998Z inflating: build/lib/libprotobuf.a 2024-08-20T21:46:06.6387483Z inflating: build/lib/libfmt.a 2024-08-20T21:46:06.6479214Z inflating: build/lib/libc10.so 2024-08-20T21:46:06.6479887Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T21:46:06.6481287Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T21:46:06.6500080Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T21:46:06.6963718Z inflating: build/lib/libprotoc.a 2024-08-20T21:46:06.6980093Z inflating: build/lib/libgmock.a 2024-08-20T21:46:06.6980578Z inflating: build/lib/libgtest_main.a 2024-08-20T21:46:06.6981051Z inflating: build/lib/libbenchmark_main.a 2024-08-20T21:46:06.6998701Z inflating: build/lib/libnnpack.a 2024-08-20T21:46:07.5943476Z inflating: build/lib/libdnnl.a 2024-08-20T21:46:07.6454393Z inflating: build/lib/libtensorpipe.a 2024-08-20T21:46:07.7412708Z inflating: build/lib/libfbgemm.a 2024-08-20T21:46:07.7680804Z inflating: build/lib/libkineto.a 2024-08-20T21:46:07.7681437Z inflating: build/lib/libgmock_main.a 2024-08-20T21:46:07.7719340Z inflating: build/lib/libonnx_proto.a 2024-08-20T21:46:07.7889871Z inflating: build/lib/libXNNPACK.a 2024-08-20T21:46:07.8531416Z inflating: build/lib/libonnx.a 2024-08-20T21:46:10.0401220Z inflating: build/lib/libtorch_cpu.so 2024-08-20T21:46:10.0401759Z inflating: build/lib/libtorch.so 2024-08-20T21:46:10.0404789Z inflating: build/lib/libunbox_lib.a 2024-08-20T21:46:10.0408244Z inflating: build/lib/libshm.so 2024-08-20T21:46:10.0426871Z inflating: build/lib/libjitbackend_test.so 2024-08-20T21:46:10.0496182Z inflating: build/lib/libtorchbind_test.so 2024-08-20T21:46:10.0519679Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T21:46:10.0542792Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T21:46:10.2307560Z inflating: build/lib/libtorch_python.so 2024-08-20T21:46:10.2340256Z inflating: build/lib/libnnapi_backend.so 2024-08-20T21:46:10.2340743Z creating: build/bin/ 2024-08-20T21:46:10.2341073Z creating: build/bin/CMakeFiles/ 2024-08-20T21:46:10.2341665Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:10.2342333Z extracting: build/bin/CMakeFiles/progress.marks 2024-08-20T21:46:10.2343063Z inflating: build/bin/Makefile 2024-08-20T21:46:10.2343506Z inflating: build/bin/cmake_install.cmake 2024-08-20T21:46:10.2344094Z inflating: build/bin/CTestTestfile.cmake 2024-08-20T21:46:10.2393688Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T21:46:10.2445352Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T21:46:10.2496456Z inflating: build/bin/c10_Device_test 2024-08-20T21:46:10.2549321Z inflating: build/bin/c10_Scalar_test 2024-08-20T21:46:10.2608256Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T21:46:10.2656013Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T21:46:10.2706320Z inflating: build/bin/c10_SymInt_test 2024-08-20T21:46:10.2759232Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T21:46:10.2815038Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T21:46:10.2868816Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T21:46:10.2917987Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T21:46:10.2971762Z inflating: build/bin/c10_Bitset_test 2024-08-20T21:46:10.3041228Z inflating: build/bin/c10_cow_test 2024-08-20T21:46:10.3090558Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T21:46:10.3139839Z inflating: build/bin/c10_Half_test 2024-08-20T21:46:10.3195189Z inflating: build/bin/c10_LeftRight_test 2024-08-20T21:46:10.3250036Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T21:46:10.3299408Z inflating: build/bin/c10_Synchronized_test 2024-08-20T21:46:10.3354100Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T21:46:10.3405451Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T21:46:10.3455237Z inflating: build/bin/c10_TypeList_test 2024-08-20T21:46:10.3506363Z inflating: build/bin/c10_accumulate_test 2024-08-20T21:46:10.3555584Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T21:46:10.3610506Z inflating: build/bin/c10_bfloat16_test 2024-08-20T21:46:10.3660603Z inflating: build/bin/c10_bit_cast_test 2024-08-20T21:46:10.3715908Z inflating: build/bin/c10_complex_math_test 2024-08-20T21:46:10.3771304Z inflating: build/bin/c10_complex_test 2024-08-20T21:46:10.3821106Z inflating: build/bin/c10_exception_test 2024-08-20T21:46:10.3872353Z inflating: build/bin/c10_flags_test 2024-08-20T21:46:10.3921228Z inflating: build/bin/c10_generic_math_test 2024-08-20T21:46:10.3975434Z inflating: build/bin/c10_lazy_test 2024-08-20T21:46:10.4024279Z inflating: build/bin/c10_irange_test 2024-08-20T21:46:10.4180012Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T21:46:10.4236412Z inflating: build/bin/c10_logging_test 2024-08-20T21:46:10.4309286Z inflating: build/bin/c10_optional_test 2024-08-20T21:46:10.4362366Z inflating: build/bin/c10_registry_test 2024-08-20T21:46:10.4422296Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T21:46:10.4569550Z inflating: build/bin/c10_small_vector_test 2024-08-20T21:46:10.4620013Z inflating: build/bin/c10_ssize_test 2024-08-20T21:46:10.4672504Z inflating: build/bin/c10_string_util_test 2024-08-20T21:46:10.4729927Z inflating: build/bin/c10_string_view_test 2024-08-20T21:46:10.4779925Z inflating: build/bin/c10_tempfile_test 2024-08-20T21:46:10.4825057Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T21:46:10.4881975Z inflating: build/bin/c10_typeid_test 2024-08-20T21:46:10.5313241Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T21:46:10.5744732Z inflating: build/bin/protoc 2024-08-20T21:46:10.6077925Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T21:46:10.6394388Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T21:46:10.6739786Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T21:46:10.6792962Z inflating: build/bin/BackoffTest 2024-08-20T21:46:10.6845513Z inflating: build/bin/FileStoreTest 2024-08-20T21:46:10.6900006Z inflating: build/bin/TCPStoreTest 2024-08-20T21:46:10.6953499Z inflating: build/bin/HashStoreTest 2024-08-20T21:46:10.6955265Z inflating: build/bin/example_allreduce 2024-08-20T21:46:10.7018933Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T21:46:10.7073801Z inflating: build/bin/test_dist_autograd 2024-08-20T21:46:10.7074642Z inflating: build/bin/parallel_benchmark 2024-08-20T21:46:10.7141523Z inflating: build/bin/test_cpp_rpc 2024-08-20T21:46:10.7149665Z inflating: build/bin/aot_model_compiler_test 2024-08-20T21:46:10.7214732Z inflating: build/bin/test_mobile_nnc 2024-08-20T21:46:10.7539077Z inflating: build/bin/test_lazy 2024-08-20T21:46:10.7596427Z inflating: build/bin/static_runtime_bench 2024-08-20T21:46:10.7665931Z inflating: build/bin/Dict_test 2024-08-20T21:46:10.7718851Z inflating: build/bin/Dimname_test 2024-08-20T21:46:10.7957860Z inflating: build/bin/static_runtime_test 2024-08-20T21:46:10.8013730Z inflating: build/bin/NamedTensor_test 2024-08-20T21:46:10.8077208Z inflating: build/bin/MaybeOwned_test 2024-08-20T21:46:10.8135227Z inflating: build/bin/apply_utils_test 2024-08-20T21:46:10.8194242Z inflating: build/bin/atest 2024-08-20T21:46:10.8254682Z inflating: build/bin/basic 2024-08-20T21:46:10.8307578Z inflating: build/bin/broadcast_test 2024-08-20T21:46:10.8358570Z inflating: build/bin/cpu_allocator_test 2024-08-20T21:46:10.8416440Z inflating: build/bin/cpu_generator_test 2024-08-20T21:46:10.8467344Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T21:46:10.8558178Z inflating: build/bin/cpu_rng_test 2024-08-20T21:46:10.8609941Z inflating: build/bin/dispatch_key_set_test 2024-08-20T21:46:10.8659094Z inflating: build/bin/dlconvertor_test 2024-08-20T21:46:10.8716347Z inflating: build/bin/extension_backend_test 2024-08-20T21:46:10.8771274Z inflating: build/bin/half_test 2024-08-20T21:46:10.9840051Z inflating: build/bin/test_api 2024-08-20T21:46:10.9889017Z inflating: build/bin/lazy_tensor_test 2024-08-20T21:46:10.9940195Z inflating: build/bin/math_kernel_test 2024-08-20T21:46:11.0033762Z inflating: build/bin/ivalue_test 2024-08-20T21:46:11.0085960Z inflating: build/bin/memory_format_test 2024-08-20T21:46:11.0136577Z inflating: build/bin/operator_name_test 2024-08-20T21:46:11.0188755Z inflating: build/bin/memory_overlapping_test 2024-08-20T21:46:11.0243529Z inflating: build/bin/native_test 2024-08-20T21:46:11.0295488Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T21:46:11.0345735Z inflating: build/bin/operators_test 2024-08-20T21:46:11.0398491Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T21:46:11.0463205Z inflating: build/bin/pow_test 2024-08-20T21:46:11.0514563Z inflating: build/bin/reduce_ops_test 2024-08-20T21:46:11.0572597Z inflating: build/bin/quantized_test 2024-08-20T21:46:11.0620695Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T21:46:11.0678968Z inflating: build/bin/scalar_tensor_test 2024-08-20T21:46:11.0729163Z inflating: build/bin/StorageUtils_test 2024-08-20T21:46:11.0786229Z inflating: build/bin/scalar_test 2024-08-20T21:46:11.0838178Z inflating: build/bin/stride_properties_test 2024-08-20T21:46:11.0915152Z inflating: build/bin/tensor_iterator_test 2024-08-20T21:46:11.0969701Z inflating: build/bin/type_ptr_test 2024-08-20T21:46:11.1022112Z inflating: build/bin/test_parallel 2024-08-20T21:46:11.1024243Z inflating: build/bin/thread_init_test 2024-08-20T21:46:11.1084130Z inflating: build/bin/type_test 2024-08-20T21:46:11.1135022Z inflating: build/bin/undefined_tensor_test 2024-08-20T21:46:11.1135707Z inflating: build/bin/verify_api_visibility 2024-08-20T21:46:11.1204124Z inflating: build/bin/legacy_vmap_test 2024-08-20T21:46:11.1255003Z inflating: build/bin/weakref_test 2024-08-20T21:46:11.1303777Z inflating: build/bin/wrapdim_test 2024-08-20T21:46:11.1362759Z inflating: build/bin/IListRef_test 2024-08-20T21:46:11.1414807Z inflating: build/bin/xla_tensor_test 2024-08-20T21:46:11.1517519Z inflating: build/bin/List_test 2024-08-20T21:46:11.1634690Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T21:46:11.1727393Z inflating: build/bin/kernel_function_test 2024-08-20T21:46:11.1848935Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T21:46:11.1914668Z inflating: build/bin/KernelFunction_test 2024-08-20T21:46:11.2011515Z inflating: build/bin/kernel_lambda_test 2024-08-20T21:46:11.2060665Z inflating: build/bin/CppSignature_test 2024-08-20T21:46:11.2121257Z inflating: build/bin/kernel_stackbased_test 2024-08-20T21:46:11.2214140Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T21:46:11.2260545Z inflating: build/bin/op_allowlist_test 2024-08-20T21:46:11.2323131Z inflating: build/bin/inline_container_test 2024-08-20T21:46:11.2618888Z inflating: build/bin/op_registration_test 2024-08-20T21:46:11.2672653Z inflating: build/bin/backend_fallback_test 2024-08-20T21:46:11.2724869Z inflating: build/bin/test_edge_op_registration 2024-08-20T21:46:11.2735636Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T21:46:11.2739733Z inflating: build/bin/torch_shm_manager 2024-08-20T21:46:11.3491861Z inflating: build/bin/test_tensorexpr 2024-08-20T21:46:11.4039416Z inflating: build/bin/test_jit 2024-08-20T21:46:11.4039870Z creating: .additional_ci_files/ 2024-08-20T21:46:11.4097408Z inflating: .additional_ci_files/test-times.json 2024-08-20T21:46:11.4326786Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T21:46:11.4426448Z ##[group]Run rm artifacts.zip 2024-08-20T21:46:11.4426827Z rm artifacts.zip 2024-08-20T21:46:11.4432480Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:11.4432939Z env: 2024-08-20T21:46:11.4433168Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.4433479Z ##[endgroup] 2024-08-20T21:46:11.4804461Z ##[group]Run df -H 2024-08-20T21:46:11.4804736Z df -H 2024-08-20T21:46:11.4809690Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:11.4810144Z env: 2024-08-20T21:46:11.4810374Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.4810691Z ##[endgroup] 2024-08-20T21:46:11.4848448Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T21:46:11.4849008Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T21:46:11.4849462Z tmpfs 50G 0 50G 0% /dev/shm 2024-08-20T21:46:11.4850178Z tmpfs 20G 652k 20G 1% /run 2024-08-20T21:46:11.4850683Z /dev/nvme0n1p1 215G 39G 177G 18% / 2024-08-20T21:46:11.4851287Z tmpfs 50G 13k 50G 1% /tmp 2024-08-20T21:46:11.4851750Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T21:46:11.4891552Z Prepare all required actions 2024-08-20T21:46:11.4891971Z Getting action download info 2024-08-20T21:46:11.6026292Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T21:46:11.6026725Z with: 2024-08-20T21:46:11.6026933Z env: 2024-08-20T21:46:11.6027173Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.6027486Z ##[endgroup] 2024-08-20T21:46:11.6223942Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:46:11.6224369Z with: 2024-08-20T21:46:11.6224610Z name: td_results 2024-08-20T21:46:11.6224882Z s3-bucket: gha-artifacts 2024-08-20T21:46:11.6225202Z region: us-east-1 2024-08-20T21:46:11.6225471Z env: 2024-08-20T21:46:11.6225715Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.6226008Z ##[endgroup] 2024-08-20T21:46:12.0891648Z (node:118037) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:46:12.0892404Z 2024-08-20T21:46:12.0892726Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:46:12.0893453Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:46:12.0894354Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:46:12.1567002Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T21:46:12.1572112Z Artifact download has finished successfully 2024-08-20T21:46:12.1797863Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T21:46:12.1798484Z mkdir -p .additional_ci_files 2024-08-20T21:46:12.1799149Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T21:46:12.1805368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:12.1805810Z env: 2024-08-20T21:46:12.1806057Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.1806369Z ##[endgroup] 2024-08-20T21:46:12.1854387Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T21:46:12.1866194Z ##[error]Process completed with exit code 1. 2024-08-20T21:46:12.2045709Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T21:46:12.2046142Z .github/scripts/parse_ref.py 2024-08-20T21:46:12.2051283Z shell: /usr/bin/bash -e {0} 2024-08-20T21:46:12.2051585Z env: 2024-08-20T21:46:12.2051828Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.2052138Z ##[endgroup] 2024-08-20T21:46:12.2339761Z Prepare all required actions 2024-08-20T21:46:12.2418053Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T21:46:12.2418616Z with: 2024-08-20T21:46:12.2419393Z github-token: *** 2024-08-20T21:46:12.2419678Z env: 2024-08-20T21:46:12.2419920Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.2420248Z ##[endgroup] 2024-08-20T21:46:12.2664516Z ##[group]Run set -eux 2024-08-20T21:46:12.2664837Z set -eux 2024-08-20T21:46:12.2665421Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T21:46:12.2671054Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:12.2671519Z env: 2024-08-20T21:46:12.2671774Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.2672271Z GITHUB_TOKEN: *** 2024-08-20T21:46:12.2672562Z ##[endgroup] 2024-08-20T21:46:12.2694842Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-07cb510f54fcb7b8e 2024-08-20T21:46:13.4331112Z setting job-id=29025550704 2024-08-20T21:46:13.4332411Z setting job-name=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:13.4585871Z Prepare all required actions 2024-08-20T21:46:13.4586325Z Getting action download info 2024-08-20T21:46:13.5668946Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T21:46:13.5669396Z with: 2024-08-20T21:46:13.5669869Z github-token: *** 2024-08-20T21:46:13.5689589Z test-matrix: {"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]} 2024-08-20T21:46:13.5710320Z job-name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:13.5711308Z env: 2024-08-20T21:46:13.5711569Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:13.5711897Z ##[endgroup] 2024-08-20T21:46:13.5864073Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:46:13.5864633Z with: 2024-08-20T21:46:13.5864891Z shell: bash 2024-08-20T21:46:13.5865184Z timeout_minutes: 10 2024-08-20T21:46:13.5865504Z max_attempts: 5 2024-08-20T21:46:13.5865794Z retry_wait_seconds: 30 2024-08-20T21:46:13.5866943Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:46:13.5868150Z polling_interval_seconds: 1 2024-08-20T21:46:13.5868518Z warning_on_retry: true 2024-08-20T21:46:13.5868846Z continue_on_error: false 2024-08-20T21:46:13.5869174Z env: 2024-08-20T21:46:13.5869438Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:13.5869956Z GITHUB_TOKEN: *** 2024-08-20T21:46:13.5870513Z ##[endgroup] 2024-08-20T21:46:13.6583739Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:46:13.8961271Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:46:13.9118244Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T21:46:13.9120090Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T21:46:13.9240817Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T21:46:13.9247609Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T21:46:13.9251711Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T21:46:13.9257830Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T21:46:14.6419425Z Command completed after 1 attempt(s). 2024-08-20T21:46:14.6493121Z ##[group]Run set -x 2024-08-20T21:46:14.6493452Z set -x 2024-08-20T21:46:14.6493854Z  2024-08-20T21:46:14.6494392Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:46:14.6495138Z # in runner workspace 2024-08-20T21:46:14.6495627Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T21:46:14.6502109Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:14.6502615Z env: 2024-08-20T21:46:14.6502871Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:14.6503210Z ##[endgroup] 2024-08-20T21:46:14.6530722Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T21:46:14.6819705Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:46:14.6820319Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:46:14.6820779Z echo "Job name: ${JOB_NAME}" 2024-08-20T21:46:14.6821173Z  2024-08-20T21:46:14.6821710Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:46:14.6822378Z # in runner workspace 2024-08-20T21:46:14.6822946Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T21:46:14.6823587Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T21:46:14.6824037Z  --job-name "${JOB_NAME}" \ 2024-08-20T21:46:14.6844115Z  --test-matrix "{"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]}" \ 2024-08-20T21:46:14.6864773Z  --selected-test-configs "" \ 2024-08-20T21:46:14.6865231Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T21:46:14.6865635Z  --tag "${TAG}" \ 2024-08-20T21:46:14.6866016Z  --event-name "${EVENT_NAME}" \ 2024-08-20T21:46:14.6866453Z  --schedule "${SCHEDULE}" \ 2024-08-20T21:46:14.6866871Z  --branch "${HEAD_BRANCH}" 2024-08-20T21:46:14.6872572Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:14.6873062Z env: 2024-08-20T21:46:14.6873323Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:14.6873857Z GITHUB_TOKEN: *** 2024-08-20T21:46:14.6874738Z JOB_NAME: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:14.6875665Z PR_NUMBER: 2024-08-20T21:46:14.6875948Z TAG: ciflow/inductor/133712 2024-08-20T21:46:14.6876282Z EVENT_NAME: push 2024-08-20T21:46:14.6876544Z SCHEDULE: 2024-08-20T21:46:14.6876799Z HEAD_BRANCH: 2024-08-20T21:46:14.6877062Z ##[endgroup] 2024-08-20T21:46:14.6901025Z Workflow: inductor 2024-08-20T21:46:14.6902235Z Job name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:14.9723630Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T21:46:15.1317792Z ##[group]Run echo "Filtered matrix:" 2024-08-20T21:46:15.1318250Z echo "Filtered matrix:" 2024-08-20T21:46:15.1337355Z echo "{"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]}" 2024-08-20T21:46:15.1357928Z  2024-08-20T21:46:15.1358176Z echo 2024-08-20T21:46:15.1358545Z echo "Is the current job unstable? False" 2024-08-20T21:46:15.1359000Z  2024-08-20T21:46:15.1359241Z echo 2024-08-20T21:46:15.1359585Z echo "Is keep-going label set? False" 2024-08-20T21:46:15.1360019Z  2024-08-20T21:46:15.1360272Z echo 2024-08-20T21:46:15.1360560Z echo "Renabled issues? " 2024-08-20T21:46:15.1367072Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:15.1367534Z env: 2024-08-20T21:46:15.1367784Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:15.1368223Z ##[endgroup] 2024-08-20T21:46:15.1391103Z Filtered matrix: 2024-08-20T21:46:15.1417879Z {include: [{config: inductor_avx512, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: inductor_avx512, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface_amp_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_timm_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_timm_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_huggingface_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_timm_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_timm_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: amz2023.linux.24xl.spr-metal}, {config: inductor_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: inductor_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_huggingface_freezing_avx2, shard: 1, num_shards: 1, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_torchbench_freezing_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_torchbench_freezing_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_timm_freezing_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_timm_freezing_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}]} 2024-08-20T21:46:15.1438468Z 2024-08-20T21:46:15.1438619Z Is the current job unstable? False 2024-08-20T21:46:15.1438869Z 2024-08-20T21:46:15.1439060Z Is keep-going label set? False 2024-08-20T21:46:15.1439288Z 2024-08-20T21:46:15.1439390Z Renabled issues? 2024-08-20T21:46:15.1589308Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:46:15.1590018Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:46:15.1595935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:15.1596430Z env: 2024-08-20T21:46:15.1596679Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:15.1597013Z JOB_TIMEOUT: 240 2024-08-20T21:46:15.1597299Z ##[endgroup] 2024-08-20T21:46:15.1835847Z ##[group]Run set -x 2024-08-20T21:46:15.1836210Z set -x 2024-08-20T21:46:15.1836470Z  2024-08-20T21:46:15.1836794Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T21:46:15.1837282Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T21:46:15.1837797Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T21:46:15.1838264Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T21:46:15.1838636Z else 2024-08-20T21:46:15.1838927Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:46:15.1839308Z fi 2024-08-20T21:46:15.1839549Z  2024-08-20T21:46:15.1839973Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T21:46:15.1840669Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T21:46:15.1841291Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T21:46:15.1841842Z # shellcheck disable=SC2086,SC2090 2024-08-20T21:46:15.1842254Z container_name=$(docker run \ 2024-08-20T21:46:15.1842632Z  ${GPU_FLAG:-} \ 2024-08-20T21:46:15.1843077Z  -e BUILD_ENVIRONMENT \ 2024-08-20T21:46:15.1843435Z  -e PR_NUMBER \ 2024-08-20T21:46:15.1843745Z  -e GITHUB_ACTIONS \ 2024-08-20T21:46:15.1844096Z  -e GITHUB_REPOSITORY \ 2024-08-20T21:46:15.1844464Z  -e GITHUB_WORKFLOW \ 2024-08-20T21:46:15.1844809Z  -e GITHUB_JOB \ 2024-08-20T21:46:15.1845120Z  -e GITHUB_RUN_ID \ 2024-08-20T21:46:15.1845465Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T21:46:15.1845826Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T21:46:15.1846176Z  -e JOB_ID \ 2024-08-20T21:46:15.1846459Z  -e JOB_NAME \ 2024-08-20T21:46:15.1846761Z  -e BASE_SHA \ 2024-08-20T21:46:15.1847068Z  -e BRANCH \ 2024-08-20T21:46:15.1847351Z  -e SHA1 \ 2024-08-20T21:46:15.1847658Z  -e AWS_DEFAULT_REGION \ 2024-08-20T21:46:15.1848021Z  -e IN_WHEEL_TEST \ 2024-08-20T21:46:15.1848523Z  -e SHARD_NUMBER \ 2024-08-20T21:46:15.1848862Z  -e TEST_CONFIG \ 2024-08-20T21:46:15.1849206Z  -e NUM_TEST_SHARDS \ 2024-08-20T21:46:15.1849574Z  -e REENABLED_ISSUES \ 2024-08-20T21:46:15.1849964Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T21:46:15.1850355Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T21:46:15.1850735Z  -e TEST_SHOWLOCALS \ 2024-08-20T21:46:15.1851101Z  -e NO_TEST_TIMEOUT \ 2024-08-20T21:46:15.1851453Z  -e NO_TD \ 2024-08-20T21:46:15.1851757Z  -e TD_DISTRIBUTED \ 2024-08-20T21:46:15.1852115Z  -e PR_LABELS \ 2024-08-20T21:46:15.1852494Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T21:46:15.1852915Z  -e SCCACHE_BUCKET \ 2024-08-20T21:46:15.1853283Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T21:46:15.1853669Z  -e XLA_CUDA \ 2024-08-20T21:46:15.1854361Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T21:46:15.1855114Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T21:46:15.1855637Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T21:46:15.1856144Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T21:46:15.1856598Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T21:46:15.1857026Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T21:46:15.1857462Z  -e DASHBOARD_TAG \ 2024-08-20T21:46:15.1857925Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T21:46:15.1858541Z  --security-opt seccomp=unconfined \ 2024-08-20T21:46:15.1858986Z  --cap-add=SYS_PTRACE \ 2024-08-20T21:46:15.1859371Z  --ipc=host \ 2024-08-20T21:46:15.1859715Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T21:46:15.1860101Z  --tty \ 2024-08-20T21:46:15.1860388Z  --detach \ 2024-08-20T21:46:15.1860876Z  --name="${container_name}" \ 2024-08-20T21:46:15.1861288Z  --user jenkins \ 2024-08-20T21:46:15.1861763Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T21:46:15.1862306Z  -w /var/lib/jenkins/workspace \ 2024-08-20T21:46:15.1862729Z  "${DOCKER_IMAGE}" 2024-08-20T21:46:15.1863064Z ) 2024-08-20T21:46:15.1863459Z # Propagate download.pytorch.org IP to container 2024-08-20T21:46:15.1864499Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T21:46:15.1865636Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T21:46:15.1866663Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T21:46:15.1872612Z shell: /usr/bin/bash -e {0} 2024-08-20T21:46:15.1872929Z env: 2024-08-20T21:46:15.1873187Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:15.1873606Z BUILD_ENVIRONMENT: linux-jammy-py3.8-gcc11-build 2024-08-20T21:46:15.1874056Z PR_NUMBER: 2024-08-20T21:46:15.1874341Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T21:46:15.1874930Z GITHUB_WORKFLOW: inductor 2024-08-20T21:46:15.1875247Z GITHUB_JOB: test 2024-08-20T21:46:15.1875528Z GITHUB_RUN_ID: 10479310725 2024-08-20T21:46:15.1875842Z GITHUB_RUN_NUMBER: 88082 2024-08-20T21:46:15.1876160Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T21:46:15.1876459Z JOB_ID: 29025550704 2024-08-20T21:46:15.1877186Z JOB_NAME: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:15.1877963Z BRANCH: 2024-08-20T21:46:15.1878269Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:46:15.1878750Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:46:15.1879236Z TEST_CONFIG: cpu_aot_inductor_torchbench_freezing 2024-08-20T21:46:15.1879640Z SHARD_NUMBER: 1 2024-08-20T21:46:15.1879910Z NUM_TEST_SHARDS: 2 2024-08-20T21:46:15.1880200Z REENABLED_ISSUES: 2024-08-20T21:46:15.1880487Z CONTINUE_THROUGH_ERROR: False 2024-08-20T21:46:15.1880836Z VERBOSE_TEST_LOGS: False 2024-08-20T21:46:15.1881152Z TEST_SHOWLOCALS: False 2024-08-20T21:46:15.1881458Z NO_TEST_TIMEOUT: False 2024-08-20T21:46:15.1881740Z NO_TD: False 2024-08-20T21:46:15.1882007Z TD_DISTRIBUTED: False 2024-08-20T21:46:15.1882388Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T21:46:15.1882842Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T21:46:15.1883171Z SHM_SIZE: 1g 2024-08-20T21:46:15.1884124Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:46:15.1885165Z XLA_CUDA: 2024-08-20T21:46:15.1885603Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:46:15.1886163Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T21:46:15.1886555Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T21:46:15.1886925Z DASHBOARD_TAG: 2024-08-20T21:46:15.1887401Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T21:46:15.1887734Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T21:46:15.1888068Z ##[endgroup] 2024-08-20T21:46:15.1910132Z + [[ cpu_aot_inductor_torchbench_freezing == \m\u\l\t\i\g\p\u ]] 2024-08-20T21:46:15.1910888Z + [[ linux-jammy-py3.8-gcc11-build == *onnx* ]] 2024-08-20T21:46:15.1911351Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:46:15.1918866Z +++ nproc --ignore=2 2024-08-20T21:46:15.2126079Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:47:32.1305368Z + container_name=dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T21:47:32.1306998Z + grep download.pytorch.org /etc/hosts 2024-08-20T21:47:32.1309059Z + docker exec -i dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T21:47:32.2691692Z + echo DOCKER_CONTAINER_ID=dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T21:47:32.2696436Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-20T21:47:32.2699095Z + docker exec -t dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T21:47:32.6422210Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T21:47:33.0685974Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T21:47:33.0688094Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T21:47:33.0690109Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T21:47:33.0692010Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T21:47:33.0695527Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T21:47:33.0700230Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.12.1) 2024-08-20T21:47:33.0718423Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T21:47:33.0733439Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T21:47:33.0747412Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.21.2) 2024-08-20T21:47:33.1117415Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T21:47:33.7620914Z Installing collected packages: torch 2024-08-20T21:47:41.1746462Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T21:47:41.1747906Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T21:47:41.1748625Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T21:47:41.2397198Z + export TERM=vt100 2024-08-20T21:47:41.2397866Z + TERM=vt100 2024-08-20T21:47:41.2398184Z ++ dirname .ci/pytorch/test.sh 2024-08-20T21:47:41.2402299Z + source .ci/pytorch/common.sh 2024-08-20T21:47:41.2405759Z +++ dirname .ci/pytorch/common.sh 2024-08-20T21:47:41.2411899Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T21:47:41.2413004Z +++ declare -f -t trap_add 2024-08-20T21:47:41.2416296Z ++ set -ex 2024-08-20T21:47:41.2416763Z ++ [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:41.2417225Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T21:47:41.2417677Z + [[ linux-jammy-py3.8-gcc11-build != *rocm* ]] 2024-08-20T21:47:41.2420630Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T21:47:41.2429905Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T21:47:41.2430370Z + trap_add cleanup_workspace EXIT 2024-08-20T21:47:41.2430773Z + trap_add_cmd=cleanup_workspace 2024-08-20T21:47:41.2431126Z + shift 2024-08-20T21:47:41.2431386Z + for trap_add_name in "$@" 2024-08-20T21:47:41.2438130Z +++ trap -p EXIT 2024-08-20T21:47:41.2439779Z ++ eval 'extract_trap_cmd ' 2024-08-20T21:47:41.2440152Z +++ extract_trap_cmd 2024-08-20T21:47:41.2440487Z +++ printf '%s\n' '' 2024-08-20T21:47:41.2440867Z ++ printf '%s\n' cleanup_workspace 2024-08-20T21:47:41.2442292Z + trap -- ' 2024-08-20T21:47:41.2442607Z cleanup_workspace' EXIT 2024-08-20T21:47:41.2443051Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T21:47:41.6792643Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T21:47:41.6809262Z + echo 'Environment variables:' 2024-08-20T21:47:41.6809686Z Environment variables: 2024-08-20T21:47:41.6809988Z + env 2024-08-20T21:47:41.6814529Z INSTALLED_DB=yes 2024-08-20T21:47:41.6815248Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:47:41.6815840Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:47:41.6816332Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11-build 2024-08-20T21:47:41.6816799Z HOSTNAME=dceb742940ef 2024-08-20T21:47:41.6818367Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.6819209Z GITHUB_ACTION=__self 2024-08-20T21:47:41.6819560Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:47:41.6819947Z GITHUB_RUN_NUMBER=88082 2024-08-20T21:47:41.6820337Z TEST_CONFIG=cpu_aot_inductor_torchbench_freezing 2024-08-20T21:47:41.6820812Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:47:41.6821285Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:47:41.6821706Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:47:41.6822160Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:41.6822573Z GITHUB_REF_TYPE=tag 2024-08-20T21:47:41.6822871Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:47:41.6823287Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.6823734Z XLA_CUDA= 2024-08-20T21:47:41.6824236Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T21:47:41.6826232Z *** 2024-08-20T21:47:41.6826497Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:47:41.6826853Z GITHUB_ACTIONS=true 2024-08-20T21:47:41.6827227Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.6827718Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.6828548Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T21:47:41.6829279Z UCC_HOME=/usr 2024-08-20T21:47:41.6829560Z VERBOSE_TEST_LOGS=False 2024-08-20T21:47:41.6829913Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T21:47:41.6830330Z SHARD_NUMBER=1 2024-08-20T21:47:41.6830622Z GITHUB_REF_PROTECTED=false 2024-08-20T21:47:41.6830960Z HOME=/var/lib/jenkins 2024-08-20T21:47:41.6831305Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:47:41.6831733Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:47:41.6832112Z UCX_COMMIT= 2024-08-20T21:47:41.6832395Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T21:47:41.6832735Z NUM_TEST_SHARDS=2 2024-08-20T21:47:41.6833056Z UCX_HOME=/usr 2024-08-20T21:47:41.6834134Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.6835584Z JOB_NAME=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:47:41.6836989Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.6838167Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:47:41.6838837Z GITHUB_EVENT_NAME=push 2024-08-20T21:47:41.6839153Z DASHBOARD_TAG= 2024-08-20T21:47:41.6839427Z GITHUB_RUN_ID=10479310725 2024-08-20T21:47:41.6840387Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.6841338Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:41.6841687Z PR_NUMBER= 2024-08-20T21:47:41.6841934Z DESIRED_CUDA= 2024-08-20T21:47:41.6842212Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:47:41.6842534Z ANACONDA_PYTHON_VERSION=3.8 2024-08-20T21:47:41.6842971Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:47:41.6843412Z TERM=vt100 2024-08-20T21:47:41.6843678Z INSTALLED_VISION=yes 2024-08-20T21:47:41.6844120Z BRANCH= 2024-08-20T21:47:41.6844378Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:47:41.6844740Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:47:41.6845535Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:47:41.6846310Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:47:41.6846690Z UCC_COMMIT= 2024-08-20T21:47:41.6846954Z REENABLED_ISSUES= 2024-08-20T21:47:41.6847235Z DOCS=yes 2024-08-20T21:47:41.6847493Z INSTALLED_ANDROID= 2024-08-20T21:47:41.6847766Z SHLVL=1 2024-08-20T21:47:41.6848007Z MAX_JOBS=46 2024-08-20T21:47:41.6848278Z GITHUB_ACTOR_ID=54816060 2024-08-20T21:47:41.6848716Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.6849251Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T21:47:41.6849927Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:47:41.6850507Z GITHUB_JOB=test 2024-08-20T21:47:41.6850786Z NO_TEST_TIMEOUT=False 2024-08-20T21:47:41.6851111Z TD_DISTRIBUTED=False 2024-08-20T21:47:41.6851447Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:47:41.6851833Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:47:41.6852156Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:47:41.6852496Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:47:41.6853573Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:41.6854944Z GITHUB_BASE_REF= 2024-08-20T21:47:41.6855226Z INSTALLED_ACL= 2024-08-20T21:47:41.6855500Z CI=true 2024-08-20T21:47:41.6855792Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:47:41.6856144Z JOB_ID=29025550704 2024-08-20T21:47:41.6856443Z INSTALLED_PROTOBUF=yes 2024-08-20T21:47:41.6856762Z GITHUB_HEAD_REF= 2024-08-20T21:47:41.6857052Z GITHUB_ACTION_REF= 2024-08-20T21:47:41.6857497Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:47:41.6857966Z TEST_SHOWLOCALS=False 2024-08-20T21:47:41.6858368Z GITHUB_WORKFLOW=inductor 2024-08-20T21:47:41.6858713Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:47:41.6859672Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.6860508Z NO_TD=False 2024-08-20T21:47:41.6860795Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:47:41.6861132Z _=/usr/bin/env 2024-08-20T21:47:41.6861615Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T21:47:41.7047482Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-08-20T21:47:41.7048446Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-20T21:47:41.7049333Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-08-20T21:47:41.7050226Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-08-20T21:47:41.7051029Z + BUILD_DIR=build 2024-08-20T21:47:41.7051356Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T21:47:41.7051726Z + BUILD_BIN_DIR=build/bin 2024-08-20T21:47:41.7052061Z + SHARD_NUMBER=1 2024-08-20T21:47:41.7052352Z + NUM_TEST_SHARDS=2 2024-08-20T21:47:41.7052655Z + export VALGRIND=ON 2024-08-20T21:47:41.7052945Z + VALGRIND=ON 2024-08-20T21:47:41.7053368Z + [[ linux-jammy-py3.8-gcc11-build == *clang9* ]] 2024-08-20T21:47:41.7053815Z + [[ 0 == \1 ]] 2024-08-20T21:47:41.7054093Z + [[ False == \1 ]] 2024-08-20T21:47:41.7054917Z + [[ linux-jammy-py3.8-gcc11-build != *bazel* ]] 2024-08-20T21:47:41.7055417Z ++ realpath build/custom_test_artifacts 2024-08-20T21:47:41.7056098Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T21:47:41.7056785Z + [[ -n '' ]] 2024-08-20T21:47:41.7057117Z + echo 'Environment variables' 2024-08-20T21:47:41.7057485Z Environment variables 2024-08-20T21:47:41.7057785Z + env 2024-08-20T21:47:41.7062701Z INSTALLED_DB=yes 2024-08-20T21:47:41.7063582Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:47:41.7064227Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:47:41.7064919Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11-build 2024-08-20T21:47:41.7065382Z HOSTNAME=dceb742940ef 2024-08-20T21:47:41.7066252Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.7067086Z GITHUB_ACTION=__self 2024-08-20T21:47:41.7067424Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:47:41.7067811Z GITHUB_RUN_NUMBER=88082 2024-08-20T21:47:41.7068189Z TEST_CONFIG=cpu_aot_inductor_torchbench_freezing 2024-08-20T21:47:41.7068661Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:47:41.7069130Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:47:41.7069551Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:47:41.7069976Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:41.7070390Z GITHUB_REF_TYPE=tag 2024-08-20T21:47:41.7070699Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:47:41.7071117Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.7071541Z XLA_CUDA= 2024-08-20T21:47:41.7072010Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T21:47:41.7072398Z *** 2024-08-20T21:47:41.7072667Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:47:41.7073008Z GITHUB_ACTIONS=true 2024-08-20T21:47:41.7073366Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.7073876Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.7074692Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T21:47:41.7075404Z UCC_HOME=/usr 2024-08-20T21:47:41.7075686Z VERBOSE_TEST_LOGS=False 2024-08-20T21:47:41.7076054Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T21:47:41.7076459Z SHARD_NUMBER=1 2024-08-20T21:47:41.7076754Z GITHUB_REF_PROTECTED=false 2024-08-20T21:47:41.7077092Z HOME=/var/lib/jenkins 2024-08-20T21:47:41.7077443Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:47:41.7077863Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:47:41.7078242Z UCX_COMMIT= 2024-08-20T21:47:41.7078521Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T21:47:41.7078878Z NUM_TEST_SHARDS=2 2024-08-20T21:47:41.7079152Z UCX_HOME=/usr 2024-08-20T21:47:41.7080021Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.7081450Z JOB_NAME=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:47:41.7082842Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.7083993Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:47:41.7084657Z GITHUB_EVENT_NAME=push 2024-08-20T21:47:41.7084973Z DASHBOARD_TAG= 2024-08-20T21:47:41.7085261Z GITHUB_RUN_ID=10479310725 2024-08-20T21:47:41.7086316Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.7087275Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:41.7087634Z PR_NUMBER= 2024-08-20T21:47:41.7087901Z DESIRED_CUDA= 2024-08-20T21:47:41.7088170Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:47:41.7088472Z VALGRIND=ON 2024-08-20T21:47:41.7088752Z ANACONDA_PYTHON_VERSION=3.8 2024-08-20T21:47:41.7089171Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:47:41.7089626Z TERM=vt100 2024-08-20T21:47:41.7089893Z INSTALLED_VISION=yes 2024-08-20T21:47:41.7090192Z BRANCH= 2024-08-20T21:47:41.7090450Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:47:41.7090811Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:47:41.7091585Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:47:41.7092356Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:47:41.7092732Z UCC_COMMIT= 2024-08-20T21:47:41.7093000Z REENABLED_ISSUES= 2024-08-20T21:47:41.7093285Z DOCS=yes 2024-08-20T21:47:41.7093533Z INSTALLED_ANDROID= 2024-08-20T21:47:41.7093819Z SHLVL=1 2024-08-20T21:47:41.7094067Z MAX_JOBS=46 2024-08-20T21:47:41.7094422Z GITHUB_ACTOR_ID=54816060 2024-08-20T21:47:41.7094864Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:41.7095401Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T21:47:41.7096078Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:47:41.7096645Z GITHUB_JOB=test 2024-08-20T21:47:41.7096933Z NO_TEST_TIMEOUT=False 2024-08-20T21:47:41.7097249Z TD_DISTRIBUTED=False 2024-08-20T21:47:41.7097583Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:47:41.7097957Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:47:41.7098467Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:47:41.7098813Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:47:41.7099887Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:41.7100975Z GITHUB_BASE_REF= 2024-08-20T21:47:41.7101260Z INSTALLED_ACL= 2024-08-20T21:47:41.7101530Z CI=true 2024-08-20T21:47:41.7101802Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:47:41.7102154Z JOB_ID=29025550704 2024-08-20T21:47:41.7102452Z INSTALLED_PROTOBUF=yes 2024-08-20T21:47:41.7102764Z GITHUB_HEAD_REF= 2024-08-20T21:47:41.7103038Z GITHUB_ACTION_REF= 2024-08-20T21:47:41.7103479Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:47:41.7103943Z TEST_SHOWLOCALS=False 2024-08-20T21:47:41.7104262Z GITHUB_WORKFLOW=inductor 2024-08-20T21:47:41.7104596Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:47:41.7105538Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_42d18a29-f060-4fcd-9b22-f2f603ca71f1 2024-08-20T21:47:41.7106371Z NO_TD=False 2024-08-20T21:47:41.7106657Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:47:41.7106993Z _=/usr/bin/env 2024-08-20T21:47:41.7107342Z + echo 'Testing pytorch' 2024-08-20T21:47:41.7107672Z Testing pytorch 2024-08-20T21:47:41.7107986Z + export LANG=C.UTF-8 2024-08-20T21:47:41.7108302Z + LANG=C.UTF-8 2024-08-20T21:47:41.7108578Z + PR_NUMBER= 2024-08-20T21:47:41.7108984Z + [[ cpu_aot_inductor_torchbench_freezing == \d\e\f\a\u\l\t ]] 2024-08-20T21:47:41.7109629Z + [[ cpu_aot_inductor_torchbench_freezing == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T21:47:41.7110264Z + [[ cpu_aot_inductor_torchbench_freezing == \s\l\o\w ]] 2024-08-20T21:47:41.7110917Z + [[ linux-jammy-py3.8-gcc11-build == *slow-gradcheck* ]] 2024-08-20T21:47:41.7111535Z + [[ linux-jammy-py3.8-gcc11-build == *cuda* ]] 2024-08-20T21:47:41.7112081Z + [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:41.7112638Z + [[ linux-jammy-py3.8-gcc11-build == *xpu* ]] 2024-08-20T21:47:41.7113182Z + [[ cpu_aot_inductor_torchbench_freezing == *crossref* ]] 2024-08-20T21:47:41.7113782Z + [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:41.7114323Z + [[ linux-jammy-py3.8-gcc11-build == *xpu* ]] 2024-08-20T21:47:41.7114882Z + [[ linux-jammy-py3.8-gcc11-build != *-bazel-* ]] 2024-08-20T21:47:41.7115495Z + pip_install --user ninja==1.10.2 2024-08-20T21:47:41.7116041Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T21:47:42.1005519Z Collecting ninja==1.10.2 2024-08-20T21:47:42.1100376Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T21:47:42.1184888Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T21:47:42.7621063Z Installing collected packages: ninja 2024-08-20T21:47:42.7703712Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T21:47:42.7705037Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T21:47:42.7849304Z Successfully installed ninja-1.10.2 2024-08-20T21:47:42.8567045Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:42.8569333Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:42.8571163Z + [[ linux-jammy-py3.8-gcc11-build == *aarch64* ]] 2024-08-20T21:47:42.8571628Z + install_tlparse 2024-08-20T21:47:42.8572006Z + pip_install --user tlparse==0.3.25 2024-08-20T21:47:42.8572565Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T21:47:43.7006252Z Collecting tlparse==0.3.25 2024-08-20T21:47:43.7132990Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T21:47:43.7688852Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T21:47:44.6141677Z Installing collected packages: tlparse 2024-08-20T21:47:44.6853720Z Successfully installed tlparse-0.3.25 2024-08-20T21:47:44.7497295Z ++ python -m site --user-base 2024-08-20T21:47:44.7761687Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:44.7763396Z + [[ linux-jammy-py3.8-gcc11-build == *asan* ]] 2024-08-20T21:47:44.7763977Z + [[ linux-jammy-py3.8-gcc11-build == *-debug* ]] 2024-08-20T21:47:44.7764570Z + [[ linux-jammy-py3.8-gcc11-build != *-bazel-* ]] 2024-08-20T21:47:44.7765428Z + echo 'We are not in debug mode: linux-jammy-py3.8-gcc11-build. Expect the assertion to pass' 2024-08-20T21:47:44.7766504Z We are not in debug mode: linux-jammy-py3.8-gcc11-build. Expect the assertion to pass 2024-08-20T21:47:44.7767157Z + cd test 2024-08-20T21:47:44.7767677Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T21:47:46.1379766Z + [[ cpu_aot_inductor_torchbench_freezing == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T21:47:46.1380579Z + [[ cpu_aot_inductor_torchbench_freezing == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T21:47:46.1381184Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T21:47:46.1381673Z + [[ cpu_aot_inductor_torchbench_freezing == *pr_time_benchmarks* ]] 2024-08-20T21:47:46.1382333Z + [[ cpu_aot_inductor_torchbench_freezing == *dynamo_eager* ]] 2024-08-20T21:47:46.1382939Z + [[ cpu_aot_inductor_torchbench_freezing == *aot_eager* ]] 2024-08-20T21:47:46.1383542Z + [[ cpu_aot_inductor_torchbench_freezing == *aot_inductor* ]] 2024-08-20T21:47:46.1384322Z + DYNAMO_BENCHMARK_FLAGS+=(--export-aot-inductor) 2024-08-20T21:47:46.1384877Z + [[ cpu_aot_inductor_torchbench_freezing == *dynamic* ]] 2024-08-20T21:47:46.1385432Z + [[ cpu_aot_inductor_torchbench_freezing == *cpu* ]] 2024-08-20T21:47:46.1385983Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2024-08-20T21:47:46.1406809Z + [[ linux-jammy-py3.8-gcc11-build == *libtorch* ]] 2024-08-20T21:47:46.1407591Z + [[ linux-jammy-py3.8-gcc11-build == *-bazel-* ]] 2024-08-20T21:47:46.1408346Z + cd test 2024-08-20T21:47:46.1408804Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T21:47:47.2977099Z PyTorch built with: 2024-08-20T21:47:47.2977720Z - GCC 11.4 2024-08-20T21:47:47.2978084Z - C++ Version: 201703 2024-08-20T21:47:47.2979105Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:47:47.2980207Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:47:47.2980874Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:47:47.2981381Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T21:47:47.2981855Z - NNPACK is enabled 2024-08-20T21:47:47.2982213Z - CPU capability usage: AVX512 2024-08-20T21:47:47.2989843Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T21:47:47.2997001Z 2024-08-20T21:47:47.4848546Z + cd test 2024-08-20T21:47:47.4849385Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T21:47:48.7455422Z ATen/Parallel: 2024-08-20T21:47:48.7455875Z at::get_num_threads() : 24 2024-08-20T21:47:48.7456291Z at::get_num_interop_threads() : 24 2024-08-20T21:47:48.7456688Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:47:48.7457076Z omp_get_max_threads() : 24 2024-08-20T21:47:48.7458260Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:47:48.7459085Z mkl_get_max_threads() : 24 2024-08-20T21:47:48.7459714Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:47:48.7460346Z std::thread::hardware_concurrency() : 48 2024-08-20T21:47:48.7460769Z Environment variables: 2024-08-20T21:47:48.7461101Z OMP_NUM_THREADS : [not set] 2024-08-20T21:47:48.7461450Z MKL_NUM_THREADS : [not set] 2024-08-20T21:47:48.7461826Z ATen parallel backend: OpenMP 2024-08-20T21:47:48.7462071Z 2024-08-20T21:47:48.9590120Z + [[ linux-jammy-py3.8-gcc11-build == *aarch64* ]] 2024-08-20T21:47:48.9590782Z + [[ cpu_aot_inductor_torchbench_freezing == *backward* ]] 2024-08-20T21:47:48.9591351Z + [[ cpu_aot_inductor_torchbench_freezing == *xla* ]] 2024-08-20T21:47:48.9591950Z + [[ cpu_aot_inductor_torchbench_freezing == *executorch* ]] 2024-08-20T21:47:48.9592581Z + [[ cpu_aot_inductor_torchbench_freezing == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T21:47:48.9593256Z + [[ linux-jammy-py3.8-gcc11-build == *libtorch* ]] 2024-08-20T21:47:48.9593828Z + [[ cpu_aot_inductor_torchbench_freezing == distributed ]] 2024-08-20T21:47:48.9594479Z + [[ cpu_aot_inductor_torchbench_freezing == *inductor_distributed* ]] 2024-08-20T21:47:48.9595233Z + [[ cpu_aot_inductor_torchbench_freezing == *inductor-halide* ]] 2024-08-20T21:47:48.9595999Z + [[ cpu_aot_inductor_torchbench_freezing == *inductor-micro-benchmark* ]] 2024-08-20T21:47:48.9596978Z + [[ cpu_aot_inductor_torchbench_freezing == *huggingface* ]] 2024-08-20T21:47:48.9597567Z + [[ cpu_aot_inductor_torchbench_freezing == *timm* ]] 2024-08-20T21:47:48.9598151Z + [[ cpu_aot_inductor_torchbench_freezing == *torchbench* ]] 2024-08-20T21:47:48.9598706Z + [[ cpu_aot_inductor_torchbench_freezing == *cpu* ]] 2024-08-20T21:47:48.9599169Z + install_torchaudio cpu 2024-08-20T21:47:48.9599494Z + local commit 2024-08-20T21:47:48.9599786Z ++ get_pinned_commit audio 2024-08-20T21:47:48.9600136Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T21:47:48.9603837Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:48.9604287Z + [[ cpu == \c\u\d\a ]] 2024-08-20T21:47:48.9605223Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:48.9606729Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:49.2828513Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:49.2832918Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-7tzlhdb9 2024-08-20T21:47:49.2866464Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-7tzlhdb9 2024-08-20T21:47:50.0106400Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T21:47:50.0152273Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:50.1014045Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:50.1015081Z Running command git submodule update --init --recursive -q 2024-08-20T21:47:52.0245354Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:47:52.0289872Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T21:47:52.0319884Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T21:47:52.0321738Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T21:47:52.0323523Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T21:47:52.0326419Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T21:47:52.0329612Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T21:47:52.0335578Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.12.1) 2024-08-20T21:47:52.0347791Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T21:47:52.0709768Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T21:47:52.0775978Z Building wheels for collected packages: torchaudio 2024-08-20T21:48:46.1782735Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T21:48:46.1841669Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp38-cp38-linux_x86_64.whl size=1767362 sha256=a8c5e59762ab98c299e897968815ddb77de8140f138182ac3febe085bd1fa60d 2024-08-20T21:48:46.1843444Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/2d/7d/0d/bdfdb3f2b184f238c1d61b3f9b248992c036dfdb76e5397d45 2024-08-20T21:48:46.1878777Z Successfully built torchaudio 2024-08-20T21:48:46.7623600Z Installing collected packages: torchaudio 2024-08-20T21:48:46.9528742Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T21:48:47.0663166Z + install_torchtext 2024-08-20T21:48:47.0663551Z + local data_commit 2024-08-20T21:48:47.0663855Z + local text_commit 2024-08-20T21:48:47.0665508Z ++ get_pinned_commit data 2024-08-20T21:48:47.0665937Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T21:48:47.0675679Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.0678797Z ++ get_pinned_commit text 2024-08-20T21:48:47.0679166Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T21:48:47.0690015Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:48:47.0691251Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.0693032Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.3901762Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.3904294Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-zbfkohrn 2024-08-20T21:48:47.3939992Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-zbfkohrn 2024-08-20T21:48:47.8612088Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T21:48:47.8647162Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.9586910Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:48.1004624Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:48.1005729Z Running command git submodule update --init --recursive -q 2024-08-20T21:49:36.4336049Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:49:36.4382795Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T21:49:36.4385518Z Requirement already satisfied: requests in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T21:49:36.4389823Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T21:49:36.4422010Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T21:49:36.4425932Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T21:49:36.4429079Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T21:49:36.4432253Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T21:49:36.4435442Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T21:49:36.4440818Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-08-20T21:49:36.4457033Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T21:49:36.4518287Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T21:49:36.4520813Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T21:49:36.4523276Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T21:49:36.4905430Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T21:49:36.4975176Z Building wheels for collected packages: torchdata 2024-08-20T21:49:38.2598716Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T21:49:38.2613945Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=46d6fc0c6f58ca1b1de79b25b5bcfe02b19cb7338bd7b69233f231337b2fa857 2024-08-20T21:49:38.2615930Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/f2/b8/b4/f1929a760cf6ea9bda553072159f9dc096899c37ad8a5ffafe 2024-08-20T21:49:38.2650260Z Successfully built torchdata 2024-08-20T21:49:38.8247953Z Installing collected packages: torchdata 2024-08-20T21:49:38.9140999Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T21:49:40.9804631Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:40.9806211Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:41.3008663Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:41.3010330Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-wriefxcp 2024-08-20T21:49:41.3044427Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-wriefxcp 2024-08-20T21:49:42.1766536Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T21:49:42.1803998Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:42.2756905Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:42.4308744Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:42.4309850Z Running command git submodule update --init --recursive -q 2024-08-20T21:49:48.7828978Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:49:48.7884018Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T21:49:48.7885665Z Requirement already satisfied: requests in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T21:49:48.7887345Z Requirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T21:49:48.7890643Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-08-20T21:49:48.7894489Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T21:49:48.7926411Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T21:49:48.7928269Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T21:49:48.7931663Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T21:49:48.7935796Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T21:49:48.7964480Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T21:49:48.7968109Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T21:49:48.7971428Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T21:49:48.7974891Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T21:49:48.7979087Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T21:49:48.7984919Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-08-20T21:49:48.8001523Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T21:49:48.8472149Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T21:49:48.8663033Z Building wheels for collected packages: torchtext 2024-08-20T21:50:17.1573843Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T21:50:17.1637724Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp38-cp38-linux_x86_64.whl size=2045014 sha256=05e7f8d688104c4a7c1fc625817dd5d3102a903b1b8c39295aa951bc050fe629 2024-08-20T21:50:17.1641030Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d6/1b/67/0243915f173a4bc67968d5441b501e3df9c77a50521995f939 2024-08-20T21:50:17.1668848Z Successfully built torchtext 2024-08-20T21:50:17.7258582Z Installing collected packages: torchtext 2024-08-20T21:50:17.8320221Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T21:50:17.9385549Z + install_torchvision 2024-08-20T21:50:17.9385960Z + local orig_preload 2024-08-20T21:50:17.9387702Z + local commit 2024-08-20T21:50:17.9388064Z ++ get_pinned_commit vision 2024-08-20T21:50:17.9388463Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T21:50:17.9398498Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:17.9398952Z + orig_preload= 2024-08-20T21:50:17.9399403Z + '[' -n '' ']' 2024-08-20T21:50:17.9400286Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:17.9401800Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:18.2615487Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:18.2617170Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-qw7f0n2t 2024-08-20T21:50:18.2651372Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-qw7f0n2t 2024-08-20T21:50:19.6615566Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T21:50:19.6651732Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:19.7555866Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:20.0276536Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:22.0510165Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:50:22.0559485Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-08-20T21:50:22.0564000Z Requirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T21:50:22.0570504Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T21:50:22.0644944Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T21:50:22.0648350Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T21:50:22.0652247Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T21:50:22.0657661Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T21:50:22.0662598Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T21:50:22.0669929Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-08-20T21:50:22.0686660Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T21:50:22.1048926Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T21:50:22.1115574Z Building wheels for collected packages: torchvision 2024-08-20T21:50:47.5337693Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T21:50:47.5379666Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp38-cp38-linux_x86_64.whl size=1215628 sha256=562d88258a83b7a60a7aecc8f2b0ea5793a79ad01e10e775fbdf044f3b9e1629 2024-08-20T21:50:47.5381420Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/82/f9/1e/f126983e23cfd567a4cb268f7d0778919a2a7014fe9af04f11 2024-08-20T21:50:47.5548879Z Successfully built torchvision 2024-08-20T21:50:48.1181200Z Installing collected packages: torchvision 2024-08-20T21:50:48.4909251Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T21:50:48.5860718Z + '[' -n '' ']' 2024-08-20T21:50:48.5861162Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T21:50:48.5861663Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T21:50:48.5862414Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T21:50:48.9157156Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T21:50:48.9160500Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-fnxicf95 2024-08-20T21:50:48.9193275Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-fnxicf95 2024-08-20T21:50:49.4898122Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T21:50:51.4166032Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:50:51.4215849Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T21:51:05.4268173Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | done 2024-08-20T21:51:05.4374859Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp38-cp38-linux_x86_64.whl size=3735232 sha256=0aec5f7b1b1583471fabd497a66dde952c5a1dee7b750cbe8fbac842b3cf7602 2024-08-20T21:51:05.4376782Z Stored in directory: /tmp/pip-ephem-wheel-cache-o0mooc5h/wheels/7a/d0/ee/d845e81513be33ad662e10ba9fb85a811ff1ad92ea52f8b6a9 2024-08-20T21:51:05.4414290Z Successfully built torchao 2024-08-20T21:51:06.0629444Z Installing collected packages: torchao 2024-08-20T21:51:06.3399827Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T21:51:06.4182331Z + id=0 2024-08-20T21:51:06.4182927Z + pip_install opencv-python==4.8.0.74 2024-08-20T21:51:06.4183542Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T21:51:06.8548533Z Collecting opencv-python==4.8.0.74 2024-08-20T21:51:06.8675292Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T21:51:06.8741653Z Requirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-08-20T21:51:06.8821321Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T21:51:08.1501215Z Installing collected packages: opencv-python 2024-08-20T21:51:09.0056780Z Successfully installed opencv-python-4.8.0.74 2024-08-20T21:51:09.0809076Z + [[ cpu_aot_inductor_torchbench_freezing == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T21:51:09.0809994Z + [[ cpu_aot_inductor_torchbench_freezing == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T21:51:09.0810832Z + [[ cpu_aot_inductor_torchbench_freezing == *torchbench_gcp_smoketest* ]] 2024-08-20T21:51:09.0811418Z + checkout_install_torchbench 2024-08-20T21:51:09.0811760Z + local commit 2024-08-20T21:51:09.0812062Z ++ get_pinned_commit torchbench 2024-08-20T21:51:09.0812479Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T21:51:09.0820452Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T21:51:09.0821053Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T21:51:09.0836379Z Cloning into 'torchbench'... 2024-08-20T21:51:09.2451848Z remote: Enumerating objects: 30997, done. 2024-08-20T21:51:09.2453451Z remote: Counting objects: 0% (1/1737) 2024-08-20T21:51:09.2455484Z remote: Counting objects: 1% (18/1737) 2024-08-20T21:51:09.2456963Z remote: Counting objects: 2% (35/1737) 2024-08-20T21:51:09.2458555Z remote: Counting objects: 3% (53/1737) 2024-08-20T21:51:09.2459086Z remote: Counting objects: 4% (70/1737) 2024-08-20T21:51:09.2459623Z remote: Counting objects: 5% (87/1737) 2024-08-20T21:51:09.2460124Z remote: Counting objects: 6% (105/1737) 2024-08-20T21:51:09.2460650Z remote: Counting objects: 7% (122/1737) 2024-08-20T21:51:09.2461181Z remote: Counting objects: 8% (139/1737) 2024-08-20T21:51:09.2461710Z remote: Counting objects: 9% (157/1737) 2024-08-20T21:51:09.2462209Z remote: Counting objects: 10% (174/1737) 2024-08-20T21:51:09.2462717Z remote: Counting objects: 11% (192/1737) 2024-08-20T21:51:09.2463230Z remote: Counting objects: 12% (209/1737) 2024-08-20T21:51:09.2463763Z remote: Counting objects: 13% (226/1737) 2024-08-20T21:51:09.2464266Z remote: Counting objects: 14% (244/1737) 2024-08-20T21:51:09.2464780Z remote: Counting objects: 15% (261/1737) 2024-08-20T21:51:09.2465297Z remote: Counting objects: 16% (278/1737) 2024-08-20T21:51:09.2465804Z remote: Counting objects: 17% (296/1737) 2024-08-20T21:51:09.2466301Z remote: Counting objects: 18% (313/1737) 2024-08-20T21:51:09.2467254Z remote: Counting objects: 19% (331/1737) 2024-08-20T21:51:09.2467777Z remote: Counting objects: 20% (348/1737) 2024-08-20T21:51:09.2468283Z remote: Counting objects: 21% (365/1737) 2024-08-20T21:51:09.2468807Z remote: Counting objects: 22% (383/1737) 2024-08-20T21:51:09.2469321Z remote: Counting objects: 23% (400/1737) 2024-08-20T21:51:09.2469830Z remote: Counting objects: 24% (417/1737) 2024-08-20T21:51:09.2470333Z remote: Counting objects: 25% (435/1737) 2024-08-20T21:51:09.2470846Z remote: Counting objects: 26% (452/1737) 2024-08-20T21:51:09.2471361Z remote: Counting objects: 27% (469/1737) 2024-08-20T21:51:09.2471874Z remote: Counting objects: 28% (487/1737) 2024-08-20T21:51:09.2472367Z remote: Counting objects: 29% (504/1737) 2024-08-20T21:51:09.2472889Z remote: Counting objects: 30% (522/1737) 2024-08-20T21:51:09.2473398Z remote: Counting objects: 31% (539/1737) 2024-08-20T21:51:09.2473906Z remote: Counting objects: 32% (556/1737) 2024-08-20T21:51:09.2474411Z remote: Counting objects: 33% (574/1737) 2024-08-20T21:51:09.2474924Z remote: Counting objects: 34% (591/1737) 2024-08-20T21:51:09.2475585Z remote: Counting objects: 35% (608/1737) 2024-08-20T21:51:09.2476103Z remote: Counting objects: 36% (626/1737) 2024-08-20T21:51:09.2476601Z remote: Counting objects: 37% (643/1737) 2024-08-20T21:51:09.2477112Z remote: Counting objects: 38% (661/1737) 2024-08-20T21:51:09.2477640Z remote: Counting objects: 39% (678/1737) 2024-08-20T21:51:09.2478148Z remote: Counting objects: 40% (695/1737) 2024-08-20T21:51:09.2478650Z remote: Counting objects: 41% (713/1737) 2024-08-20T21:51:09.2479186Z remote: Counting objects: 42% (730/1737) 2024-08-20T21:51:09.2479698Z remote: Counting objects: 43% (747/1737) 2024-08-20T21:51:09.2480209Z remote: Counting objects: 44% (765/1737) 2024-08-20T21:51:09.2480705Z remote: Counting objects: 45% (782/1737) 2024-08-20T21:51:09.2481214Z remote: Counting objects: 46% (800/1737) 2024-08-20T21:51:09.2481728Z remote: Counting objects: 47% (817/1737) 2024-08-20T21:51:09.2482236Z remote: Counting objects: 48% (834/1737) 2024-08-20T21:51:09.2482735Z remote: Counting objects: 49% (852/1737) 2024-08-20T21:51:09.2483246Z remote: Counting objects: 50% (869/1737) 2024-08-20T21:51:09.2483757Z remote: Counting objects: 51% (886/1737) 2024-08-20T21:51:09.2484266Z remote: Counting objects: 52% (904/1737) 2024-08-20T21:51:09.2484763Z remote: Counting objects: 53% (921/1737) 2024-08-20T21:51:09.2485270Z remote: Counting objects: 54% (938/1737) 2024-08-20T21:51:09.2485782Z remote: Counting objects: 55% (956/1737) 2024-08-20T21:51:09.2486278Z remote: Counting objects: 56% (973/1737) 2024-08-20T21:51:09.2486785Z remote: Counting objects: 57% (991/1737) 2024-08-20T21:51:09.2487300Z remote: Counting objects: 58% (1008/1737) 2024-08-20T21:51:09.2487823Z remote: Counting objects: 59% (1025/1737) 2024-08-20T21:51:09.2488335Z remote: Counting objects: 60% (1043/1737) 2024-08-20T21:51:09.2488860Z remote: Counting objects: 61% (1060/1737) 2024-08-20T21:51:09.2489376Z remote: Counting objects: 62% (1077/1737) 2024-08-20T21:51:09.2489901Z remote: Counting objects: 63% (1095/1737) 2024-08-20T21:51:09.2490409Z remote: Counting objects: 64% (1112/1737) 2024-08-20T21:51:09.2490927Z remote: Counting objects: 65% (1130/1737) 2024-08-20T21:51:09.2491443Z remote: Counting objects: 66% (1147/1737) 2024-08-20T21:51:09.2491958Z remote: Counting objects: 67% (1164/1737) 2024-08-20T21:51:09.2492462Z remote: Counting objects: 68% (1182/1737) 2024-08-20T21:51:09.2492976Z remote: Counting objects: 69% (1199/1737) 2024-08-20T21:51:09.2493493Z remote: Counting objects: 70% (1216/1737) 2024-08-20T21:51:09.2494009Z remote: Counting objects: 71% (1234/1737) 2024-08-20T21:51:09.2494511Z remote: Counting objects: 72% (1251/1737) 2024-08-20T21:51:09.2495026Z remote: Counting objects: 73% (1269/1737) 2024-08-20T21:51:09.2496606Z remote: Counting objects: 74% (1286/1737) 2024-08-20T21:51:09.2497165Z remote: Counting objects: 75% (1303/1737) 2024-08-20T21:51:09.2497669Z remote: Counting objects: 76% (1321/1737) 2024-08-20T21:51:09.2498202Z remote: Counting objects: 77% (1338/1737) 2024-08-20T21:51:09.2498854Z remote: Counting objects: 78% (1355/1737) 2024-08-20T21:51:09.2499359Z remote: Counting objects: 79% (1373/1737) 2024-08-20T21:51:09.2499881Z remote: Counting objects: 80% (1390/1737) 2024-08-20T21:51:09.2500397Z remote: Counting objects: 81% (1407/1737) 2024-08-20T21:51:09.2500913Z remote: Counting objects: 82% (1425/1737) 2024-08-20T21:51:09.2501416Z remote: Counting objects: 83% (1442/1737) 2024-08-20T21:51:09.2501952Z remote: Counting objects: 84% (1460/1737) 2024-08-20T21:51:09.2502451Z remote: Counting objects: 85% (1477/1737) 2024-08-20T21:51:09.2502970Z remote: Counting objects: 86% (1494/1737) 2024-08-20T21:51:09.2503487Z remote: Counting objects: 87% (1512/1737) 2024-08-20T21:51:09.2504006Z remote: Counting objects: 88% (1529/1737) 2024-08-20T21:51:09.2504509Z remote: Counting objects: 89% (1546/1737) 2024-08-20T21:51:09.2505123Z remote: Counting objects: 90% (1564/1737) 2024-08-20T21:51:09.2505640Z remote: Counting objects: 91% (1581/1737) 2024-08-20T21:51:09.2506160Z remote: Counting objects: 92% (1599/1737) 2024-08-20T21:51:09.2506663Z remote: Counting objects: 93% (1616/1737) 2024-08-20T21:51:09.2507180Z remote: Counting objects: 94% (1633/1737) 2024-08-20T21:51:09.2507695Z remote: Counting objects: 95% (1651/1737) 2024-08-20T21:51:09.2508214Z remote: Counting objects: 96% (1668/1737) 2024-08-20T21:51:09.2508717Z remote: Counting objects: 97% (1685/1737) 2024-08-20T21:51:09.2509233Z remote: Counting objects: 98% (1703/1737) 2024-08-20T21:51:09.2509752Z remote: Counting objects: 99% (1720/1737) 2024-08-20T21:51:09.2510268Z remote: Counting objects: 100% (1737/1737) 2024-08-20T21:51:09.2510813Z remote: Counting objects: 100% (1737/1737), done. 2024-08-20T21:51:09.2511387Z remote: Compressing objects: 0% (1/896) 2024-08-20T21:51:09.2576266Z remote: Compressing objects: 1% (9/896) 2024-08-20T21:51:09.2696271Z remote: Compressing objects: 2% (18/896) 2024-08-20T21:51:09.2777392Z remote: Compressing objects: 3% (27/896) 2024-08-20T21:51:09.2838586Z remote: Compressing objects: 4% (36/896) 2024-08-20T21:51:09.2888122Z remote: Compressing objects: 5% (45/896) 2024-08-20T21:51:09.2915911Z remote: Compressing objects: 6% (54/896) 2024-08-20T21:51:09.2940916Z remote: Compressing objects: 7% (63/896) 2024-08-20T21:51:09.2982151Z remote: Compressing objects: 8% (72/896) 2024-08-20T21:51:09.3016236Z remote: Compressing objects: 9% (81/896) 2024-08-20T21:51:09.3044799Z remote: Compressing objects: 10% (90/896) 2024-08-20T21:51:09.3058694Z remote: Compressing objects: 11% (99/896) 2024-08-20T21:51:09.3074319Z remote: Compressing objects: 12% (108/896) 2024-08-20T21:51:09.3081546Z remote: Compressing objects: 13% (117/896) 2024-08-20T21:51:09.3101853Z remote: Compressing objects: 14% (126/896) 2024-08-20T21:51:09.3111949Z remote: Compressing objects: 15% (135/896) 2024-08-20T21:51:09.3116080Z remote: Compressing objects: 16% (144/896) 2024-08-20T21:51:09.3119174Z remote: Compressing objects: 17% (153/896) 2024-08-20T21:51:09.3122505Z remote: Compressing objects: 18% (162/896) 2024-08-20T21:51:09.3126812Z remote: Compressing objects: 19% (171/896) 2024-08-20T21:51:09.3128694Z remote: Compressing objects: 20% (180/896) 2024-08-20T21:51:09.3130256Z remote: Compressing objects: 21% (189/896) 2024-08-20T21:51:09.3131810Z remote: Compressing objects: 22% (198/896) 2024-08-20T21:51:09.3134078Z remote: Compressing objects: 23% (207/896) 2024-08-20T21:51:09.3135744Z remote: Compressing objects: 24% (216/896) 2024-08-20T21:51:09.3136280Z remote: Compressing objects: 25% (224/896) 2024-08-20T21:51:09.3138900Z remote: Compressing objects: 26% (233/896) 2024-08-20T21:51:09.3145944Z remote: Compressing objects: 27% (242/896) 2024-08-20T21:51:09.3148801Z remote: Compressing objects: 28% (251/896) 2024-08-20T21:51:09.3152275Z remote: Compressing objects: 29% (260/896) 2024-08-20T21:51:09.3156017Z remote: Compressing objects: 30% (269/896) 2024-08-20T21:51:09.3157962Z remote: Compressing objects: 31% (278/896) 2024-08-20T21:51:09.3165025Z remote: Compressing objects: 32% (287/896) 2024-08-20T21:51:09.3166100Z remote: Compressing objects: 33% (296/896) 2024-08-20T21:51:09.3167985Z remote: Compressing objects: 34% (305/896) 2024-08-20T21:51:09.3170535Z remote: Compressing objects: 35% (314/896) 2024-08-20T21:51:09.3173158Z remote: Compressing objects: 36% (323/896) 2024-08-20T21:51:09.3175573Z remote: Compressing objects: 37% (332/896) 2024-08-20T21:51:09.3180106Z remote: Compressing objects: 38% (341/896) 2024-08-20T21:51:09.3180663Z remote: Compressing objects: 39% (350/896) 2024-08-20T21:51:09.3183002Z remote: Compressing objects: 40% (359/896) 2024-08-20T21:51:09.3183542Z remote: Compressing objects: 41% (368/896) 2024-08-20T21:51:09.3184241Z remote: Compressing objects: 42% (377/896) 2024-08-20T21:51:09.3185451Z remote: Compressing objects: 43% (386/896) 2024-08-20T21:51:09.3185998Z remote: Compressing objects: 44% (395/896) 2024-08-20T21:51:09.3186544Z remote: Compressing objects: 45% (404/896) 2024-08-20T21:51:09.3187078Z remote: Compressing objects: 46% (413/896) 2024-08-20T21:51:09.3187600Z remote: Compressing objects: 47% (422/896) 2024-08-20T21:51:09.3188139Z remote: Compressing objects: 48% (431/896) 2024-08-20T21:51:09.3188685Z remote: Compressing objects: 49% (440/896) 2024-08-20T21:51:09.3189223Z remote: Compressing objects: 50% (448/896) 2024-08-20T21:51:09.3189740Z remote: Compressing objects: 51% (457/896) 2024-08-20T21:51:09.3190271Z remote: Compressing objects: 52% (466/896) 2024-08-20T21:51:09.3190806Z remote: Compressing objects: 53% (475/896) 2024-08-20T21:51:09.3191333Z remote: Compressing objects: 54% (484/896) 2024-08-20T21:51:09.3191868Z remote: Compressing objects: 55% (493/896) 2024-08-20T21:51:09.3192416Z remote: Compressing objects: 56% (502/896) 2024-08-20T21:51:09.3192946Z remote: Compressing objects: 57% (511/896) 2024-08-20T21:51:09.3193466Z remote: Compressing objects: 58% (520/896) 2024-08-20T21:51:09.3193999Z remote: Compressing objects: 59% (529/896) 2024-08-20T21:51:09.3194532Z remote: Compressing objects: 60% (538/896) 2024-08-20T21:51:09.3195067Z remote: Compressing objects: 61% (547/896) 2024-08-20T21:51:09.3195585Z remote: Compressing objects: 62% (556/896) 2024-08-20T21:51:09.3196120Z remote: Compressing objects: 63% (565/896) 2024-08-20T21:51:09.3196649Z remote: Compressing objects: 64% (574/896) 2024-08-20T21:51:09.3197185Z remote: Compressing objects: 65% (583/896) 2024-08-20T21:51:09.3197704Z remote: Compressing objects: 66% (592/896) 2024-08-20T21:51:09.3198241Z remote: Compressing objects: 67% (601/896) 2024-08-20T21:51:09.3198776Z remote: Compressing objects: 68% (610/896) 2024-08-20T21:51:09.3199314Z remote: Compressing objects: 69% (619/896) 2024-08-20T21:51:09.3199834Z remote: Compressing objects: 70% (628/896) 2024-08-20T21:51:09.3200370Z remote: Compressing objects: 71% (637/896) 2024-08-20T21:51:09.3200905Z remote: Compressing objects: 72% (646/896) 2024-08-20T21:51:09.3201441Z remote: Compressing objects: 73% (655/896) 2024-08-20T21:51:09.3201960Z remote: Compressing objects: 74% (664/896) 2024-08-20T21:51:09.3202491Z remote: Compressing objects: 75% (672/896) 2024-08-20T21:51:09.3209090Z remote: Compressing objects: 76% (681/896) 2024-08-20T21:51:09.3216120Z remote: Compressing objects: 77% (690/896) 2024-08-20T21:51:09.3221201Z remote: Compressing objects: 78% (699/896) 2024-08-20T21:51:09.3228358Z remote: Compressing objects: 79% (708/896) 2024-08-20T21:51:09.3233931Z remote: Compressing objects: 80% (717/896) 2024-08-20T21:51:09.3243177Z remote: Compressing objects: 81% (726/896) 2024-08-20T21:51:09.3250491Z remote: Compressing objects: 82% (735/896) 2024-08-20T21:51:09.3254562Z remote: Compressing objects: 83% (744/896) 2024-08-20T21:51:09.3257548Z remote: Compressing objects: 84% (753/896) 2024-08-20T21:51:09.3262353Z remote: Compressing objects: 85% (762/896) 2024-08-20T21:51:09.3267075Z remote: Compressing objects: 86% (771/896) 2024-08-20T21:51:09.3271779Z remote: Compressing objects: 87% (780/896) 2024-08-20T21:51:09.3275364Z remote: Compressing objects: 88% (789/896) 2024-08-20T21:51:09.3278714Z remote: Compressing objects: 89% (798/896) 2024-08-20T21:51:09.3282316Z remote: Compressing objects: 90% (807/896) 2024-08-20T21:51:09.3285335Z remote: Compressing objects: 91% (816/896) 2024-08-20T21:51:09.3289123Z remote: Compressing objects: 92% (825/896) 2024-08-20T21:51:09.3292211Z remote: Compressing objects: 93% (834/896) 2024-08-20T21:51:09.3294863Z remote: Compressing objects: 94% (843/896) 2024-08-20T21:51:09.3297500Z remote: Compressing objects: 95% (852/896) 2024-08-20T21:51:09.3301558Z remote: Compressing objects: 96% (861/896) 2024-08-20T21:51:09.3303449Z remote: Compressing objects: 97% (870/896) 2024-08-20T21:51:09.3304503Z remote: Compressing objects: 98% (879/896) 2024-08-20T21:51:09.3305526Z remote: Compressing objects: 99% (888/896) 2024-08-20T21:51:09.3306069Z remote: Compressing objects: 100% (896/896) 2024-08-20T21:51:09.3306628Z remote: Compressing objects: 100% (896/896), done. 2024-08-20T21:51:09.3358074Z Receiving objects: 0% (1/30997) 2024-08-20T21:51:09.3399269Z Receiving objects: 1% (310/30997) 2024-08-20T21:51:09.3433072Z Receiving objects: 2% (620/30997) 2024-08-20T21:51:09.3464618Z Receiving objects: 3% (930/30997) 2024-08-20T21:51:09.3497197Z Receiving objects: 4% (1240/30997) 2024-08-20T21:51:09.3528426Z Receiving objects: 5% (1550/30997) 2024-08-20T21:51:09.3560492Z Receiving objects: 6% (1860/30997) 2024-08-20T21:51:09.3599514Z Receiving objects: 7% (2170/30997) 2024-08-20T21:51:09.3623395Z Receiving objects: 8% (2480/30997) 2024-08-20T21:51:09.3656135Z Receiving objects: 9% (2790/30997) 2024-08-20T21:51:09.3722733Z Receiving objects: 10% (3100/30997) 2024-08-20T21:51:09.3764779Z Receiving objects: 11% (3410/30997) 2024-08-20T21:51:09.3827599Z Receiving objects: 12% (3720/30997) 2024-08-20T21:51:09.3956569Z Receiving objects: 13% (4030/30997) 2024-08-20T21:51:09.4088102Z Receiving objects: 14% (4340/30997) 2024-08-20T21:51:09.4155588Z Receiving objects: 15% (4650/30997) 2024-08-20T21:51:09.4222311Z Receiving objects: 16% (4960/30997) 2024-08-20T21:51:09.4281989Z Receiving objects: 17% (5270/30997) 2024-08-20T21:51:09.4303847Z Receiving objects: 18% (5580/30997) 2024-08-20T21:51:09.4357841Z Receiving objects: 19% (5890/30997) 2024-08-20T21:51:09.4388508Z Receiving objects: 20% (6200/30997) 2024-08-20T21:51:09.4405843Z Receiving objects: 21% (6510/30997) 2024-08-20T21:51:09.4543480Z Receiving objects: 22% (6820/30997) 2024-08-20T21:51:10.2820634Z Receiving objects: 23% (7130/30997) 2024-08-20T21:51:10.3318971Z Receiving objects: 24% (7440/30997), 43.27 MiB | 86.53 MiB/s 2024-08-20T21:51:10.7573049Z Receiving objects: 24% (7510/30997), 91.36 MiB | 91.35 MiB/s 2024-08-20T21:51:10.8179031Z Receiving objects: 25% (7750/30997), 91.36 MiB | 91.35 MiB/s 2024-08-20T21:51:10.8781705Z Receiving objects: 26% (8060/30997), 91.36 MiB | 91.35 MiB/s 2024-08-20T21:51:10.9381898Z Receiving objects: 27% (8370/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:10.9982575Z Receiving objects: 28% (8680/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:11.0587709Z Receiving objects: 29% (8990/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:11.1194505Z Receiving objects: 30% (9300/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:11.1798413Z Receiving objects: 31% (9610/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:11.3319206Z Receiving objects: 32% (9920/30997), 131.75 MiB | 87.89 MiB/s 2024-08-20T21:51:11.3821668Z Receiving objects: 32% (10173/30997), 177.73 MiB | 88.90 MiB/s 2024-08-20T21:51:11.4504375Z Receiving objects: 33% (10230/30997), 177.73 MiB | 88.90 MiB/s 2024-08-20T21:51:11.4621039Z Receiving objects: 34% (10539/30997), 177.73 MiB | 88.90 MiB/s 2024-08-20T21:51:11.6938368Z Receiving objects: 35% (10849/30997), 177.73 MiB | 88.90 MiB/s 2024-08-20T21:51:11.9020600Z Receiving objects: 36% (11159/30997), 177.73 MiB | 88.90 MiB/s 2024-08-20T21:51:11.9037622Z Receiving objects: 37% (11469/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9056551Z Receiving objects: 38% (11779/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9074869Z Receiving objects: 39% (12089/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9085612Z Receiving objects: 40% (12399/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9094132Z Receiving objects: 41% (12709/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9105504Z Receiving objects: 42% (13019/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9123986Z Receiving objects: 43% (13329/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9139195Z Receiving objects: 44% (13639/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9141510Z Receiving objects: 45% (13949/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9145411Z Receiving objects: 46% (14259/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9158467Z Receiving objects: 47% (14569/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9180694Z Receiving objects: 48% (14879/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9189747Z Receiving objects: 49% (15189/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9379946Z Receiving objects: 50% (15499/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9381667Z Receiving objects: 51% (15809/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9387817Z Receiving objects: 52% (16119/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9393200Z Receiving objects: 53% (16429/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9397880Z Receiving objects: 54% (16739/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9417826Z Receiving objects: 55% (17049/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9423421Z Receiving objects: 56% (17359/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9428855Z Receiving objects: 57% (17669/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9445682Z Receiving objects: 58% (17979/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9452061Z Receiving objects: 59% (18289/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9461540Z Receiving objects: 60% (18599/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9580023Z Receiving objects: 61% (18909/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9581037Z Receiving objects: 62% (19219/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9586044Z Receiving objects: 63% (19529/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9593012Z Receiving objects: 64% (19839/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9599864Z Receiving objects: 65% (20149/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9607590Z Receiving objects: 66% (20459/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:11.9826124Z Receiving objects: 67% (20768/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0203291Z Receiving objects: 68% (21078/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0204921Z Receiving objects: 69% (21388/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0220464Z Receiving objects: 70% (21698/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0227451Z Receiving objects: 71% (22008/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0244270Z Receiving objects: 72% (22318/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0248702Z Receiving objects: 73% (22628/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0261719Z Receiving objects: 74% (22938/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0272476Z Receiving objects: 75% (23248/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0278859Z Receiving objects: 76% (23558/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0285714Z Receiving objects: 77% (23868/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0494236Z Receiving objects: 78% (24178/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0525318Z Receiving objects: 79% (24488/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0529102Z Receiving objects: 80% (24798/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0539918Z Receiving objects: 81% (25108/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0549600Z Receiving objects: 82% (25418/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0600228Z Receiving objects: 83% (25728/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0716256Z Receiving objects: 84% (26038/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0716957Z Receiving objects: 85% (26348/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0723174Z Receiving objects: 86% (26658/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0727790Z Receiving objects: 87% (26968/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0734209Z Receiving objects: 88% (27278/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0739212Z Receiving objects: 89% (27588/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0848496Z Receiving objects: 90% (27898/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0858258Z Receiving objects: 91% (28208/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0868704Z Receiving objects: 92% (28518/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0888554Z Receiving objects: 93% (28828/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0919067Z Receiving objects: 94% (29138/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0925257Z Receiving objects: 95% (29448/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0930995Z Receiving objects: 96% (29758/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.0936497Z Receiving objects: 97% (30068/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.1003486Z Receiving objects: 98% (30378/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.1025910Z Receiving objects: 99% (30688/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.1026981Z remote: Total 30997 (delta 1011), reused 1473 (delta 822), pack-reused 29260 (from 1) 2024-08-20T21:51:12.1047512Z Receiving objects: 100% (30997/30997), 214.77 MiB | 85.94 MiB/s 2024-08-20T21:51:12.1048228Z Receiving objects: 100% (30997/30997), 232.50 MiB | 83.86 MiB/s, done. 2024-08-20T21:51:12.1076410Z Resolving deltas: 0% (0/16731) 2024-08-20T21:51:12.1078802Z Resolving deltas: 1% (168/16731) 2024-08-20T21:51:12.1081356Z Resolving deltas: 2% (335/16731) 2024-08-20T21:51:12.1087124Z Resolving deltas: 3% (502/16731) 2024-08-20T21:51:12.1090503Z Resolving deltas: 4% (670/16731) 2024-08-20T21:51:12.1092708Z Resolving deltas: 5% (837/16731) 2024-08-20T21:51:12.1095210Z Resolving deltas: 6% (1004/16731) 2024-08-20T21:51:12.1099195Z Resolving deltas: 7% (1172/16731) 2024-08-20T21:51:12.1102632Z Resolving deltas: 8% (1339/16731) 2024-08-20T21:51:12.1106396Z Resolving deltas: 9% (1506/16731) 2024-08-20T21:51:12.1109671Z Resolving deltas: 10% (1674/16731) 2024-08-20T21:51:12.1113390Z Resolving deltas: 11% (1841/16731) 2024-08-20T21:51:12.1116715Z Resolving deltas: 12% (2009/16731) 2024-08-20T21:51:12.1120774Z Resolving deltas: 13% (2176/16731) 2024-08-20T21:51:12.1124411Z Resolving deltas: 14% (2343/16731) 2024-08-20T21:51:12.1128123Z Resolving deltas: 15% (2511/16731) 2024-08-20T21:51:12.1131985Z Resolving deltas: 16% (2677/16731) 2024-08-20T21:51:12.1135348Z Resolving deltas: 17% (2845/16731) 2024-08-20T21:51:12.1138232Z Resolving deltas: 18% (3012/16731) 2024-08-20T21:51:12.1142835Z Resolving deltas: 19% (3180/16731) 2024-08-20T21:51:12.1148096Z Resolving deltas: 20% (3347/16731) 2024-08-20T21:51:12.1154637Z Resolving deltas: 21% (3514/16731) 2024-08-20T21:51:12.1162642Z Resolving deltas: 22% (3681/16731) 2024-08-20T21:51:12.1170140Z Resolving deltas: 23% (3849/16731) 2024-08-20T21:51:12.1174822Z Resolving deltas: 24% (4016/16731) 2024-08-20T21:51:12.1182894Z Resolving deltas: 25% (4184/16731) 2024-08-20T21:51:12.1186465Z Resolving deltas: 26% (4351/16731) 2024-08-20T21:51:12.1190296Z Resolving deltas: 27% (4518/16731) 2024-08-20T21:51:12.1199755Z Resolving deltas: 28% (4685/16731) 2024-08-20T21:51:12.1204191Z Resolving deltas: 29% (4852/16731) 2024-08-20T21:51:12.1206715Z Resolving deltas: 30% (5022/16731) 2024-08-20T21:51:12.1209189Z Resolving deltas: 31% (5189/16731) 2024-08-20T21:51:12.1211682Z Resolving deltas: 32% (5354/16731) 2024-08-20T21:51:12.1213869Z Resolving deltas: 33% (5524/16731) 2024-08-20T21:51:12.1216673Z Resolving deltas: 34% (5689/16731) 2024-08-20T21:51:12.1219541Z Resolving deltas: 35% (5856/16731) 2024-08-20T21:51:12.1222870Z Resolving deltas: 36% (6024/16731) 2024-08-20T21:51:12.1226418Z Resolving deltas: 37% (6194/16731) 2024-08-20T21:51:12.1231685Z Resolving deltas: 38% (6358/16731) 2024-08-20T21:51:12.1238839Z Resolving deltas: 39% (6526/16731) 2024-08-20T21:51:12.1246692Z Resolving deltas: 40% (6693/16731) 2024-08-20T21:51:12.1252643Z Resolving deltas: 41% (6860/16731) 2024-08-20T21:51:12.1257458Z Resolving deltas: 42% (7028/16731) 2024-08-20T21:51:12.1259936Z Resolving deltas: 43% (7195/16731) 2024-08-20T21:51:12.1262796Z Resolving deltas: 44% (7363/16731) 2024-08-20T21:51:12.1265375Z Resolving deltas: 45% (7530/16731) 2024-08-20T21:51:12.1272771Z Resolving deltas: 46% (7697/16731) 2024-08-20T21:51:12.1289699Z Resolving deltas: 47% (7864/16731) 2024-08-20T21:51:12.1292927Z Resolving deltas: 48% (8032/16731) 2024-08-20T21:51:12.1296396Z Resolving deltas: 49% (8200/16731) 2024-08-20T21:51:12.1304436Z Resolving deltas: 50% (8366/16731) 2024-08-20T21:51:12.1311793Z Resolving deltas: 51% (8533/16731) 2024-08-20T21:51:12.1316639Z Resolving deltas: 52% (8701/16731) 2024-08-20T21:51:12.1321008Z Resolving deltas: 53% (8868/16731) 2024-08-20T21:51:12.1324616Z Resolving deltas: 54% (9035/16731) 2024-08-20T21:51:12.1328276Z Resolving deltas: 55% (9203/16731) 2024-08-20T21:51:12.1331961Z Resolving deltas: 56% (9370/16731) 2024-08-20T21:51:12.1335922Z Resolving deltas: 57% (9538/16731) 2024-08-20T21:51:12.1340868Z Resolving deltas: 58% (9704/16731) 2024-08-20T21:51:12.1347582Z Resolving deltas: 59% (9872/16731) 2024-08-20T21:51:12.1352981Z Resolving deltas: 60% (10039/16731) 2024-08-20T21:51:12.1359076Z Resolving deltas: 61% (10206/16731) 2024-08-20T21:51:12.1365204Z Resolving deltas: 62% (10374/16731) 2024-08-20T21:51:12.1373001Z Resolving deltas: 63% (10541/16731) 2024-08-20T21:51:12.1380935Z Resolving deltas: 64% (10708/16731) 2024-08-20T21:51:12.1387887Z Resolving deltas: 65% (10876/16731) 2024-08-20T21:51:12.1391715Z Resolving deltas: 66% (11043/16731) 2024-08-20T21:51:12.1397004Z Resolving deltas: 67% (11210/16731) 2024-08-20T21:51:12.1399990Z Resolving deltas: 68% (11379/16731) 2024-08-20T21:51:12.1406188Z Resolving deltas: 69% (11545/16731) 2024-08-20T21:51:12.1411244Z Resolving deltas: 70% (11712/16731) 2024-08-20T21:51:12.1415674Z Resolving deltas: 71% (11880/16731) 2024-08-20T21:51:12.1419185Z Resolving deltas: 72% (12047/16731) 2024-08-20T21:51:12.1426006Z Resolving deltas: 73% (12214/16731) 2024-08-20T21:51:12.1433559Z Resolving deltas: 74% (12381/16731) 2024-08-20T21:51:12.1438126Z Resolving deltas: 75% (12549/16731) 2024-08-20T21:51:12.1443003Z Resolving deltas: 76% (12716/16731) 2024-08-20T21:51:12.1446450Z Resolving deltas: 77% (12883/16731) 2024-08-20T21:51:12.1453800Z Resolving deltas: 78% (13051/16731) 2024-08-20T21:51:12.1459788Z Resolving deltas: 79% (13218/16731) 2024-08-20T21:51:12.1465757Z Resolving deltas: 80% (13385/16731) 2024-08-20T21:51:12.1471479Z Resolving deltas: 81% (13553/16731) 2024-08-20T21:51:12.1477737Z Resolving deltas: 82% (13721/16731) 2024-08-20T21:51:12.1484295Z Resolving deltas: 83% (13887/16731) 2024-08-20T21:51:12.1490533Z Resolving deltas: 84% (14055/16731) 2024-08-20T21:51:12.1495220Z Resolving deltas: 85% (14222/16731) 2024-08-20T21:51:12.1500010Z Resolving deltas: 86% (14389/16731) 2024-08-20T21:51:12.1502583Z Resolving deltas: 87% (14556/16731) 2024-08-20T21:51:12.1506629Z Resolving deltas: 88% (14724/16731) 2024-08-20T21:51:12.1511902Z Resolving deltas: 89% (14891/16731) 2024-08-20T21:51:12.1518006Z Resolving deltas: 90% (15058/16731) 2024-08-20T21:51:12.1522005Z Resolving deltas: 91% (15226/16731) 2024-08-20T21:51:12.1525923Z Resolving deltas: 92% (15394/16731) 2024-08-20T21:51:12.1529463Z Resolving deltas: 93% (15560/16731) 2024-08-20T21:51:12.1535487Z Resolving deltas: 94% (15729/16731) 2024-08-20T21:51:12.1541070Z Resolving deltas: 95% (15895/16731) 2024-08-20T21:51:12.1547437Z Resolving deltas: 96% (16062/16731) 2024-08-20T21:51:12.1554021Z Resolving deltas: 97% (16230/16731) 2024-08-20T21:51:12.1558793Z Resolving deltas: 98% (16397/16731) 2024-08-20T21:51:12.1678577Z Resolving deltas: 99% (16565/16731) 2024-08-20T21:51:12.1679022Z Resolving deltas: 100% (16731/16731) 2024-08-20T21:51:13.1306963Z Resolving deltas: 100% (16731/16731), done. 2024-08-20T21:51:13.1307471Z + pushd torchbench 2024-08-20T21:51:13.1307807Z ~/workspace/torchbench ~/workspace 2024-08-20T21:51:13.1308291Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T21:51:13.1904186Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T21:51:13.1904644Z 2024-08-20T21:51:13.1905046Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T21:51:13.1905834Z changes and commit them, and you can discard any commits you make in this 2024-08-20T21:51:13.1906593Z state without impacting any branches by switching back to a branch. 2024-08-20T21:51:13.1907031Z 2024-08-20T21:51:13.1907332Z If you want to create a new branch to retain commits you create, you may 2024-08-20T21:51:13.1908129Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T21:51:13.1908567Z 2024-08-20T21:51:13.1908747Z git switch -c 2024-08-20T21:51:13.1909012Z 2024-08-20T21:51:13.1909148Z Or undo this operation with: 2024-08-20T21:51:13.1909398Z 2024-08-20T21:51:13.1909500Z git switch - 2024-08-20T21:51:13.1909680Z 2024-08-20T21:51:13.1910010Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T21:51:13.1910526Z 2024-08-20T21:51:13.1910676Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T21:51:13.1911107Z + '[' '' ']' 2024-08-20T21:51:13.1911439Z + python install.py --continue_on_fail 2024-08-20T21:51:15.2067375Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T21:51:30.0287355Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T21:51:31.6625354Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T21:51:33.5708909Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T21:51:35.1942141Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T21:51:35.2244860Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T21:51:41.5947530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T21:51:44.1038861Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T21:51:46.5999050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T21:51:49.1032766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T21:51:49.1035158Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T21:51:50.1924055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T21:51:52.4211200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T21:51:52.4516360Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T21:52:35.6085310Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T21:52:43.1285704Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T21:52:49.6025234Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T21:52:55.5483491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T21:53:03.1343302Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T21:53:09.1736829Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T21:53:15.9510464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T21:53:22.2921912Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T21:53:28.8000079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T21:53:35.8486787Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T21:53:41.8702114Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T21:53:48.4479069Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T21:53:49.6530994Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T21:54:22.3107466Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T21:54:26.9755822Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T21:54:31.9652567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:00:16.3069808Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:00:17.4983491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:00:18.9590039Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:00:25.2218802Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:00:32.0428951Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:00:37.2681999Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:00:44.7339351Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:00:50.3801309Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:00:55.1422113Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:01:01.4331781Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:01:17.4049925Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:01:23.3536766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:01:27.0160323Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:01:31.5627967Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:01:38.1297050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:01:38.1298996Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:01:51.1091542Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:01:54.8027197Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:01:54.8030652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:02:02.1700761Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:02:03.4129091Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:02:04.6680292Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:02:48.5111536Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:04:16.1841788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:04:16.2157050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:04:17.6941347Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:04:17.7252569Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:04:17.7535058Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:04:17.7807191Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:04:17.8073466Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:04:17.8341547Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:04:38.6220921Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:04:38.6223672Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:04:39.8808186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:04:42.2682876Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:04:42.2990069Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:04:42.3278823Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:04:42.3553988Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:04:42.3823732Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:04:42.4096296Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:04:47.9895952Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:04:49.4534671Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:04:52.7959911Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:04:52.8270176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:04:52.8563327Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:04:52.8841653Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:04:52.9107219Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:04:52.9380163Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:05:04.0777985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:05:17.1193327Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:05:17.1510335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:05:17.1513484Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:05:17.1515635Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:05:19.9857126Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:05:22.0410231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:05:22.0726513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:10:24.5826916Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:10:38.5708177Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:10:41.3579478Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:10:47.0764318Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:10:47.1083387Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:10:47.1384335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:10:47.1665216Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:10:47.1937556Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:10:47.2210804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:10:47.2478421Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:10:47.2744907Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:10:54.3550178Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:11:00.3062111Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:11:00.3378739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:11:01.9234989Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:11:03.8396650Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:11:04.1984957Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:11:04.1985659Z Print all dependencies after TorchBench is installed 2024-08-20T22:11:04.1986192Z + python -mpip freeze 2024-08-20T22:11:04.6517246Z absl-py==2.1.0 2024-08-20T22:11:04.6517707Z accelerate==0.33.0 2024-08-20T22:11:04.6518029Z aiohappyeyeballs==2.4.0 2024-08-20T22:11:04.6518350Z aiohttp==3.10.5 2024-08-20T22:11:04.6524386Z aiosignal==1.3.1 2024-08-20T22:11:04.6524742Z alabaster==0.7.13 2024-08-20T22:11:04.6525292Z annotated-types==0.7.0 2024-08-20T22:11:04.6525712Z antlr4-python3-runtime==4.9.3 2024-08-20T22:11:04.6526077Z anyascii==0.3.2 2024-08-20T22:11:04.6526359Z astroid==3.2.4 2024-08-20T22:11:04.6527687Z asttokens==2.4.1 2024-08-20T22:11:04.6528002Z astunparse==1.6.3 2024-08-20T22:11:04.6528362Z async-timeout==4.0.3 2024-08-20T22:11:04.6528688Z attrs==23.1.0 2024-08-20T22:11:04.6528945Z audioread==3.0.1 2024-08-20T22:11:04.6529228Z babel==2.16.0 2024-08-20T22:11:04.6529494Z backcall==0.2.0 2024-08-20T22:11:04.6529770Z beautifulsoup4==4.12.3 2024-08-20T22:11:04.6530920Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:11:04.6531960Z bibtexparser==2.0.0b7 2024-08-20T22:11:04.6532267Z black==24.8.0 2024-08-20T22:11:04.6532522Z blinker==1.8.2 2024-08-20T22:11:04.6532792Z blis==0.7.11 2024-08-20T22:11:04.6533057Z blobfile==2.1.1 2024-08-20T22:11:04.6533329Z bokeh==3.1.1 2024-08-20T22:11:04.6533666Z boto3==1.19.12 2024-08-20T22:11:04.6533936Z botocore==1.22.12 2024-08-20T22:11:04.6534207Z breathe==4.34.0 2024-08-20T22:11:04.6534489Z bs4==0.0.1 2024-08-20T22:11:04.6534748Z cachetools==5.5.0 2024-08-20T22:11:04.6535036Z cardboardlint==1.3.1 2024-08-20T22:11:04.6535339Z catalogue==2.0.10 2024-08-20T22:11:04.6535636Z certifi==2024.7.4 2024-08-20T22:11:04.6536057Z cffi==1.17.0 2024-08-20T22:11:04.6536386Z charset-normalizer==3.3.2 2024-08-20T22:11:04.6536718Z click==8.1.7 2024-08-20T22:11:04.6536989Z clldutils==3.22.2 2024-08-20T22:11:04.6537270Z cloudpathlib==0.18.1 2024-08-20T22:11:04.6537576Z cloudpickle==3.0.0 2024-08-20T22:11:04.6537866Z colorama==0.4.6 2024-08-20T22:11:04.6538146Z colorlog==6.8.2 2024-08-20T22:11:04.6538528Z comm==0.2.2 2024-08-20T22:11:04.6538807Z confection==0.1.5 2024-08-20T22:11:04.6539095Z contourpy==1.1.1 2024-08-20T22:11:04.6539385Z coremltools==5.0b5 2024-08-20T22:11:04.6539668Z cryptography==43.0.0 2024-08-20T22:11:04.6539967Z csvw==3.3.0 2024-08-20T22:11:04.6540224Z cycler==0.12.1 2024-08-20T22:11:04.6540480Z cymem==2.0.8 2024-08-20T22:11:04.6540788Z DALL-E==0.1 2024-08-20T22:11:04.6541046Z datasets==2.21.0 2024-08-20T22:11:04.6541326Z debugpy==1.8.5 2024-08-20T22:11:04.6541596Z decorator==5.1.1 2024-08-20T22:11:04.6541884Z defusedxml==0.7.1 2024-08-20T22:11:04.6542174Z Deprecated==1.2.14 2024-08-20T22:11:04.6542899Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:11:04.6543709Z diffusers==0.20.2 2024-08-20T22:11:04.6543997Z dill==0.3.8 2024-08-20T22:11:04.6544258Z diskcache==5.6.3 2024-08-20T22:11:04.6544524Z distro==1.9.0 2024-08-20T22:11:04.6544789Z dlinfo==1.2.1 2024-08-20T22:11:04.6545056Z dnspython==2.6.1 2024-08-20T22:11:04.6545374Z docker-pycreds==0.4.0 2024-08-20T22:11:04.6545685Z docutils==0.16 2024-08-20T22:11:04.6545961Z dominate==2.9.1 2024-08-20T22:11:04.6546775Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:11:04.6547553Z einops==0.8.0 2024-08-20T22:11:04.6547836Z exceptiongroup==1.2.2 2024-08-20T22:11:04.6548143Z execnet==2.1.1 2024-08-20T22:11:04.6548416Z executing==2.0.1 2024-08-20T22:11:04.6548688Z exhale==0.2.3 2024-08-20T22:11:04.6548960Z expecttest==0.1.6 2024-08-20T22:11:04.6549254Z fastjsonschema==2.20.0 2024-08-20T22:11:04.6549561Z FastNLP==0.6.0 2024-08-20T22:11:04.6549884Z ffmpeg-python==0.2.0 2024-08-20T22:11:04.6550191Z filelock==3.13.1 2024-08-20T22:11:04.6550470Z Flask==3.0.3 2024-08-20T22:11:04.6550729Z flatbuffers==2.0 2024-08-20T22:11:04.6551012Z fonttools==4.53.1 2024-08-20T22:11:04.6551296Z frozenlist==1.4.1 2024-08-20T22:11:04.6551567Z fsspec==2024.6.1 2024-08-20T22:11:04.6551841Z ftfy==6.2.3 2024-08-20T22:11:04.6552451Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:11:04.6553166Z future==1.0.0 2024-08-20T22:11:04.6553435Z fvcore==0.1.5.post20221221 2024-08-20T22:11:04.6553757Z gdown==5.2.0 2024-08-20T22:11:04.6554019Z geojson==2.5.0 2024-08-20T22:11:04.6554465Z ghstack==0.8.0 2024-08-20T22:11:04.6554727Z gitdb==4.0.11 2024-08-20T22:11:04.6555003Z GitPython==3.1.43 2024-08-20T22:11:04.6555338Z google-auth==2.34.0 2024-08-20T22:11:04.6555826Z google-auth-oauthlib==1.0.0 2024-08-20T22:11:04.6556176Z greenlet==3.0.3 2024-08-20T22:11:04.6556456Z grpcio==1.65.5 2024-08-20T22:11:04.6556740Z gym==0.26.2 2024-08-20T22:11:04.6557027Z gym-notices==0.0.8 2024-08-20T22:11:04.6557313Z h5py==3.11.0 2024-08-20T22:11:04.6557574Z higher==0.2.1 2024-08-20T22:11:04.6557885Z huggingface-hub==0.24.6 2024-08-20T22:11:04.6558241Z hydra-core==1.3.2 2024-08-20T22:11:04.6558531Z hypothesis==5.35.1 2024-08-20T22:11:04.6558817Z idna==3.7 2024-08-20T22:11:04.6559055Z imageio==2.35.1 2024-08-20T22:11:04.6559332Z imagesize==1.4.1 2024-08-20T22:11:04.6559629Z importlib_metadata==8.3.0 2024-08-20T22:11:04.6559961Z importlib_resources==6.4.3 2024-08-20T22:11:04.6560294Z inflect==7.3.1 2024-08-20T22:11:04.6560574Z iniconfig==2.0.0 2024-08-20T22:11:04.6560857Z iopath==0.1.9 2024-08-20T22:11:04.6561118Z ipykernel==6.29.5 2024-08-20T22:11:04.6561407Z ipython==8.12.0 2024-08-20T22:11:04.6561681Z isodate==0.6.1 2024-08-20T22:11:04.6561940Z isort==5.13.2 2024-08-20T22:11:04.6562217Z itsdangerous==2.2.0 2024-08-20T22:11:04.6562509Z jedi==0.19.1 2024-08-20T22:11:04.6562770Z Jinja2==3.1.4 2024-08-20T22:11:04.6563113Z jmespath==0.10.0 2024-08-20T22:11:04.6563402Z joblib==1.4.2 2024-08-20T22:11:04.6563672Z jsonpatch==1.33 2024-08-20T22:11:04.6563945Z jsonpointer==3.0.0 2024-08-20T22:11:04.6564241Z jsonschema==4.23.0 2024-08-20T22:11:04.6564628Z jsonschema-specifications==2023.12.1 2024-08-20T22:11:04.6565030Z junitparser==2.1.1 2024-08-20T22:11:04.6565354Z jupyter-cache==0.6.1 2024-08-20T22:11:04.6565674Z jupyter_client==8.6.2 2024-08-20T22:11:04.6565993Z jupyter_core==5.7.2 2024-08-20T22:11:04.6566312Z kaldi-io==0.9.8 2024-08-20T22:11:04.6566595Z kiwisolver==1.4.5 2024-08-20T22:11:04.6566882Z kornia==0.7.3 2024-08-20T22:11:04.6567150Z kornia_rs==0.1.5 2024-08-20T22:11:04.6567421Z lameenc==1.7.0 2024-08-20T22:11:04.6567701Z langcodes==3.4.0 2024-08-20T22:11:04.6567989Z langdetect==1.0.9 2024-08-20T22:11:04.6568318Z language-tags==1.2.0 2024-08-20T22:11:04.6568623Z language_data==1.2.0 2024-08-20T22:11:04.6568918Z lark==0.12.0 2024-08-20T22:11:04.6569179Z libcst==1.1.0 2024-08-20T22:11:04.6569435Z librosa==0.9.2 2024-08-20T22:11:04.6569718Z lintrunner==0.12.5 2024-08-20T22:11:04.6570005Z llvmlite==0.32.1 2024-08-20T22:11:04.6570284Z lxml==4.9.4 2024-08-20T22:11:04.6570569Z marisa-trie==1.2.0 2024-08-20T22:11:04.6570856Z Markdown==3.7 2024-08-20T22:11:04.6571168Z markdown-it-py==2.2.0 2024-08-20T22:11:04.6571469Z MarkupSafe==2.1.5 2024-08-20T22:11:04.6571758Z matplotlib==3.5.3 2024-08-20T22:11:04.6572098Z matplotlib-inline==0.1.7 2024-08-20T22:11:04.6572424Z mccabe==0.7.0 2024-08-20T22:11:04.6572729Z mdit-py-plugins==0.3.5 2024-08-20T22:11:04.6573037Z mdurl==0.1.2 2024-08-20T22:11:04.6573334Z ml-dtypes==0.2.0 2024-08-20T22:11:04.6573609Z MonkeyType==23.3.0 2024-08-20T22:11:04.6573947Z more-itertools==10.4.0 2024-08-20T22:11:04.6574255Z mpmath==1.3.0 2024-08-20T22:11:04.6574526Z multidict==6.0.5 2024-08-20T22:11:04.6574807Z multiprocess==0.70.16 2024-08-20T22:11:04.6575125Z murmurhash==1.0.10 2024-08-20T22:11:04.6575412Z musdb==0.4.2 2024-08-20T22:11:04.6575663Z museval==0.4.1 2024-08-20T22:11:04.6575933Z mypy==1.10.0 2024-08-20T22:11:04.6576253Z mypy-extensions==1.0.0 2024-08-20T22:11:04.6576598Z myst-nb==0.17.2 2024-08-20T22:11:04.6576902Z myst-parser==0.18.1 2024-08-20T22:11:04.6577198Z nbclient==0.7.4 2024-08-20T22:11:04.6577473Z nbformat==5.10.4 2024-08-20T22:11:04.6577780Z nest-asyncio==1.6.0 2024-08-20T22:11:04.6578074Z networkx==2.8.8 2024-08-20T22:11:04.6578346Z ninja==1.10.2 2024-08-20T22:11:04.6578728Z nose==1.3.7 2024-08-20T22:11:04.6578972Z numba==0.49.0 2024-08-20T22:11:04.6579235Z numpy==1.21.2 2024-08-20T22:11:04.6579582Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:11:04.6579990Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:11:04.6580429Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:11:04.6580873Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:11:04.6581310Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:11:04.6581695Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:11:04.6582187Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:11:04.6582616Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:11:04.6583057Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:11:04.6583465Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:11:04.6583858Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:11:04.6584266Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:11:04.6584585Z oauthlib==3.2.2 2024-08-20T22:11:04.6584865Z omegaconf==2.3.0 2024-08-20T22:11:04.6585144Z onnx==1.16.1 2024-08-20T22:11:04.6585429Z onnxscript==0.1.0.dev20240817 2024-08-20T22:11:04.6585757Z opacus==1.5.2 2024-08-20T22:11:04.6586081Z opencv-python==4.8.0.74 2024-08-20T22:11:04.6586441Z opt-einsum==3.3.0 2024-08-20T22:11:04.6586715Z optree==0.12.1 2024-08-20T22:11:04.6586989Z packaging==24.1 2024-08-20T22:11:04.6587268Z pandas==2.0.3 2024-08-20T22:11:04.6587542Z parso==0.8.4 2024-08-20T22:11:04.6587792Z patch==1.16 2024-08-20T22:11:04.6588059Z pathspec==0.12.1 2024-08-20T22:11:04.6588345Z pexpect==4.9.0 2024-08-20T22:11:04.6588625Z phonemizer==3.3.0 2024-08-20T22:11:04.6588911Z pickleshare==0.7.5 2024-08-20T22:11:04.6589204Z pillow==10.3.0 2024-08-20T22:11:04.6589495Z pkgutil_resolve_name==1.3.10 2024-08-20T22:11:04.6589909Z platformdirs==4.2.2 2024-08-20T22:11:04.6590205Z pluggy==1.5.0 2024-08-20T22:11:04.6590481Z pooch==1.8.2 2024-08-20T22:11:04.6590757Z portalocker==2.10.1 2024-08-20T22:11:04.6591149Z preshed==3.0.9 2024-08-20T22:11:04.6591425Z prettytable==3.11.0 2024-08-20T22:11:04.6591720Z prompt_toolkit==3.0.47 2024-08-20T22:11:04.6592010Z protobuf==3.20.2 2024-08-20T22:11:04.6592284Z psutil==6.0.0 2024-08-20T22:11:04.6592549Z ptyprocess==0.7.0 2024-08-20T22:11:04.6592832Z pure_eval==0.2.3 2024-08-20T22:11:04.6593143Z py-cpuinfo==9.0.0 2024-08-20T22:11:04.6593420Z pyaml==24.7.0 2024-08-20T22:11:04.6593793Z pyarrow==17.0.0 2024-08-20T22:11:04.6594042Z pyasn1==0.6.0 2024-08-20T22:11:04.6594293Z pyasn1_modules==0.4.0 2024-08-20T22:11:04.6594586Z pyclipper==1.3.0.post5 2024-08-20T22:11:04.6594878Z pycocotools==2.0.7 2024-08-20T22:11:04.6595140Z pycparser==2.22 2024-08-20T22:11:04.6595416Z pycryptodomex==3.20.0 2024-08-20T22:11:04.6595709Z pydantic==2.8.2 2024-08-20T22:11:04.6595961Z pydantic_core==2.20.1 2024-08-20T22:11:04.6596247Z pydot==3.0.1 2024-08-20T22:11:04.6596492Z pygame==2.6.0 2024-08-20T22:11:04.6596745Z PyGithub==2.3.0 2024-08-20T22:11:04.6596992Z Pygments==2.15.0 2024-08-20T22:11:04.6597252Z PyJWT==2.9.0 2024-08-20T22:11:04.6597557Z pylatexenc==2.10 2024-08-20T22:11:04.6597818Z pylint==3.2.6 2024-08-20T22:11:04.6598053Z PyNaCl==1.5.0 2024-08-20T22:11:04.6598301Z pynvml==11.5.3 2024-08-20T22:11:04.6598560Z pyparsing==3.1.2 2024-08-20T22:11:04.6598824Z pypdfium2==4.30.0 2024-08-20T22:11:04.6599075Z pysbd==0.3.4 2024-08-20T22:11:04.6599320Z PySocks==1.7.1 2024-08-20T22:11:04.6599570Z pytest==7.3.2 2024-08-20T22:11:04.6599868Z pytest-benchmark==4.0.0 2024-08-20T22:11:04.6600205Z pytest-cpp==2.3.0 2024-08-20T22:11:04.6600524Z pytest-flakefinder==1.1.0 2024-08-20T22:11:04.6600890Z pytest-rerunfailures==14.0 2024-08-20T22:11:04.6601228Z pytest-xdist==3.3.1 2024-08-20T22:11:04.6601562Z python-dateutil==2.9.0.post0 2024-08-20T22:11:04.6602325Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:11:04.6603042Z python-etcd==0.4.5 2024-08-20T22:11:04.6604030Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:11:04.6605733Z -e git+https://github.com/pytorch/pytorch_sphinx_theme.git@05ee809ecf4c396cce8f44a172484f2946eb255a#egg=pytorch_sphinx_theme 2024-08-20T22:11:04.6606774Z pytz==2024.1 2024-08-20T22:11:04.6607048Z PyWavelets==1.4.1 2024-08-20T22:11:04.6607393Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:11:04.6607826Z pyzmq==26.1.1 2024-08-20T22:11:04.6608095Z rapidfuzz==3.9.6 2024-08-20T22:11:04.6608375Z rdflib==7.0.0 2024-08-20T22:11:04.6608621Z redis==5.0.8 2024-08-20T22:11:04.6608891Z referencing==0.35.1 2024-08-20T22:11:04.6609181Z regex==2024.7.24 2024-08-20T22:11:04.6609553Z requests==2.32.3 2024-08-20T22:11:04.6609910Z requests-oauthlib==2.0.0 2024-08-20T22:11:04.6610239Z resampy==0.3.1 2024-08-20T22:11:04.6610514Z rfc3986==1.5.0 2024-08-20T22:11:04.6610766Z rich==13.7.1 2024-08-20T22:11:04.6611027Z rockset==1.0.3 2024-08-20T22:11:04.6611330Z rpds-py==0.20.0 2024-08-20T22:11:04.6611588Z rsa==4.9 2024-08-20T22:11:04.6611842Z s3transfer==0.5.2 2024-08-20T22:11:04.6612130Z safetensors==0.4.4 2024-08-20T22:11:04.6612459Z scikit-image==0.19.3 2024-08-20T22:11:04.6612786Z scikit-learn==1.3.2 2024-08-20T22:11:04.6613078Z scipy==1.10.1 2024-08-20T22:11:04.6613346Z segments==2.2.1 2024-08-20T22:11:04.6613620Z sentencepiece==0.2.0 2024-08-20T22:11:04.6613957Z sentry-sdk==2.13.0 2024-08-20T22:11:04.6614248Z setproctitle==1.3.3 2024-08-20T22:11:04.6614543Z shapely==2.0.6 2024-08-20T22:11:04.6614806Z shellingham==1.5.4 2024-08-20T22:11:04.6615099Z simplejson==3.19.3 2024-08-20T22:11:04.6615477Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:11:04.6615954Z smart-open==7.0.4 2024-08-20T22:11:04.6616242Z smmap==5.0.1 2024-08-20T22:11:04.6616523Z snowballstemmer==2.2.0 2024-08-20T22:11:04.6616855Z sortedcontainers==2.4.0 2024-08-20T22:11:04.6617224Z soundfile==0.12.1 2024-08-20T22:11:04.6617508Z soupsieve==2.6 2024-08-20T22:11:04.6617781Z spacy==3.7.6 2024-08-20T22:11:04.6618079Z spacy-legacy==3.0.12 2024-08-20T22:11:04.6618509Z spacy-loggers==1.0.5 2024-08-20T22:11:04.6618818Z Sphinx==5.3.0 2024-08-20T22:11:04.6619155Z sphinx-copybutton==0.5.0 2024-08-20T22:11:04.6619569Z sphinx-panels==0.4.1 2024-08-20T22:11:04.6619953Z sphinxcontrib-applehelp==1.0.4 2024-08-20T22:11:04.6620385Z sphinxcontrib-devhelp==1.0.2 2024-08-20T22:11:04.6620804Z sphinxcontrib-htmlhelp==2.0.1 2024-08-20T22:11:04.6621210Z sphinxcontrib-jsmath==1.0.1 2024-08-20T22:11:04.6621612Z sphinxcontrib-katex==0.8.6 2024-08-20T22:11:04.6622009Z sphinxcontrib-qthelp==1.0.3 2024-08-20T22:11:04.6622437Z sphinxcontrib-serializinghtml==1.1.5 2024-08-20T22:11:04.6622842Z SQLAlchemy==2.0.32 2024-08-20T22:11:04.6623134Z srsly==2.4.8 2024-08-20T22:11:04.6623435Z stack-data==0.6.3 2024-08-20T22:11:04.6623706Z stempeg==0.2.3 2024-08-20T22:11:04.6623986Z submitit==1.5.1 2024-08-20T22:11:04.6624267Z sympy==1.12.1 2024-08-20T22:11:04.6624522Z tabulate==0.9.0 2024-08-20T22:11:04.6624849Z tb-nightly==2.13.0a20230426 2024-08-20T22:11:04.6625182Z tensorboard==2.13.0 2024-08-20T22:11:04.6625551Z tensorboard-data-server==0.7.2 2024-08-20T22:11:04.6625903Z tensorboardX==2.6.2.2 2024-08-20T22:11:04.6626209Z termcolor==2.4.0 2024-08-20T22:11:04.6626488Z thinc==8.2.5 2024-08-20T22:11:04.6626746Z threadpoolctl==3.5.0 2024-08-20T22:11:04.6627049Z tifffile==2023.7.10 2024-08-20T22:11:04.6627877Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:11:04.6628672Z tlparse==0.3.25 2024-08-20T22:11:04.6628942Z tokenizers==0.15.2 2024-08-20T22:11:04.6629226Z tomli==2.0.1 2024-08-20T22:11:04.6629491Z tomlkit==0.13.2 2024-08-20T22:11:04.6630762Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp38-cp38-linux_x86_64.whl#sha256=af3cdc81a563c847a86d810c93c14c6dabe8cf56a0cb55c23bba116e1c3deca6 2024-08-20T22:11:04.6632474Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:11:04.6633488Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:11:04.6634415Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:11:04.6635361Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:11:04.6636425Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:11:04.6637466Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:11:04.6638408Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:11:04.6639134Z tornado==6.4.1 2024-08-20T22:11:04.6639392Z tqdm==4.66.5 2024-08-20T22:11:04.6639626Z traitlets==5.14.3 2024-08-20T22:11:04.6639905Z transformers==4.38.1 2024-08-20T22:11:04.6640187Z treetable==0.2.5 2024-08-20T22:11:04.6640947Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:11:04.6641691Z typeguard==4.3.0 2024-08-20T22:11:04.6641952Z typer==0.12.4 2024-08-20T22:11:04.6642249Z typing-inspect==0.9.0 2024-08-20T22:11:04.6642536Z typing_extensions==4.12.2 2024-08-20T22:11:04.6642837Z tzdata==2024.1 2024-08-20T22:11:04.6643098Z Unidecode==1.3.8 2024-08-20T22:11:04.6643429Z unittest-xml-reporting==3.2.0 2024-08-20T22:11:04.6643748Z uritemplate==4.1.1 2024-08-20T22:11:04.6644015Z urllib3==1.26.19 2024-08-20T22:11:04.6644273Z visdom==0.2.4 2024-08-20T22:11:04.6644508Z wandb==0.17.7 2024-08-20T22:11:04.6644749Z wasabi==1.1.3 2024-08-20T22:11:04.6644998Z wcwidth==0.2.13 2024-08-20T22:11:04.6645259Z weasel==0.4.1 2024-08-20T22:11:04.6645544Z websocket-client==1.8.0 2024-08-20T22:11:04.6645849Z Werkzeug==3.0.3 2024-08-20T22:11:04.6646103Z wrapt==1.16.0 2024-08-20T22:11:04.6646398Z xdoctest==1.1.0 2024-08-20T22:11:04.6646655Z xxhash==3.5.0 2024-08-20T22:11:04.6646910Z xyzservices==2024.6.0 2024-08-20T22:11:04.6647186Z yacs==0.1.8 2024-08-20T22:11:04.6647408Z yarl==1.9.4 2024-08-20T22:11:04.6647686Z z3-solver==4.12.2.0 2024-08-20T22:11:04.6647954Z zipp==3.20.0 2024-08-20T22:11:04.6837481Z + popd 2024-08-20T22:11:04.6837797Z ~/workspace 2024-08-20T22:11:04.6838279Z + [[ cpu_aot_inductor_torchbench_freezing != *cpu* ]] 2024-08-20T22:11:04.6839049Z ++ pwd 2024-08-20T22:11:04.6840924Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T22:11:04.6841415Z + test_dynamo_benchmark torchbench 0 2024-08-20T22:11:04.6844858Z ++ pwd 2024-08-20T22:11:04.6847498Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:11:04.6848087Z + local suite=torchbench 2024-08-20T22:11:04.6848408Z + shift 2024-08-20T22:11:04.6848668Z + local shard_id=0 2024-08-20T22:11:04.6848934Z + shift 2024-08-20T22:11:04.6849306Z + [[ cpu_aot_inductor_torchbench_freezing == *perf_compare* ]] 2024-08-20T22:11:04.6849895Z + [[ cpu_aot_inductor_torchbench_freezing == *perf* ]] 2024-08-20T22:11:04.6850421Z + [[ cpu_aot_inductor_torchbench_freezing == *cpu* ]] 2024-08-20T22:11:04.6850872Z + local dt=float32 2024-08-20T22:11:04.6851239Z + [[ cpu_aot_inductor_torchbench_freezing == *amp* ]] 2024-08-20T22:11:04.6851796Z + [[ cpu_aot_inductor_torchbench_freezing == *freezing* ]] 2024-08-20T22:11:04.6852609Z + test_single_dynamo_benchmark inference torchbench 0 --inference --float32 --freezing 2024-08-20T22:11:04.6853242Z ++ pwd 2024-08-20T22:11:04.6853692Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:11:04.6854539Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:11:04.6866158Z + local name=inference 2024-08-20T22:11:04.6866465Z + shift 2024-08-20T22:11:04.6866735Z + local suite=torchbench 2024-08-20T22:11:04.6867048Z + shift 2024-08-20T22:11:04.6867280Z + local shard_id=0 2024-08-20T22:11:04.6867557Z + shift 2024-08-20T22:11:04.6867814Z + partition_flags=() 2024-08-20T22:11:04.6868126Z + local partition_flags 2024-08-20T22:11:04.6868464Z + [[ -n 2 ]] 2024-08-20T22:11:04.6868747Z + [[ -n 0 ]] 2024-08-20T22:11:04.6869342Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:11:04.6870085Z + [[ cpu_aot_inductor_torchbench_freezing == *perf_compare* ]] 2024-08-20T22:11:04.6870668Z + [[ cpu_aot_inductor_torchbench_freezing == *perf* ]] 2024-08-20T22:11:04.6871253Z + [[ cpu_aot_inductor_torchbench_freezing == *aot_inductor* ]] 2024-08-20T22:11:04.6871890Z + [[ cpu_aot_inductor_torchbench_freezing != *cpu_aot_inductor* ]] 2024-08-20T22:11:04.6872476Z + [[ cpu_aot_inductor_torchbench_freezing == *_avx2* ]] 2024-08-20T22:11:04.6873035Z + [[ cpu_aot_inductor_torchbench_freezing == *_avx512* ]] 2024-08-20T22:11:04.6875112Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --export-aot-inductor --device cpu --inference --float32 --freezing --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T22:11:09.7130478Z 2024-08-20T22:11:11.5284503Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:11:11.5285080Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:11:11.5302511Z cpu eval BERT_pytorch 2024-08-20T22:11:11.9400801Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:12.1170980Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:12.2916215Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:12.7046425Z ERROR:common: 2024-08-20T22:11:12.7046916Z Traceback (most recent call last): 2024-08-20T22:11:12.7047781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:11:12.7048606Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:11:12.7049654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:11:12.7050704Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:11:12.7051564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:11:12.7052299Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:11:12.7053482Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:11:12.7054553Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:11:12.7055506Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:11:12.7056330Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:11:12.7057364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:11:12.7058263Z return self._call_impl(*args, **kwargs) 2024-08-20T22:11:12.7059316Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:11:12.7060165Z return forward_call(*args, **kwargs) 2024-08-20T22:11:12.7061094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:11:12.7061866Z return fn(*args, **kwargs) 2024-08-20T22:11:12.7062803Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:11:12.7063712Z return self._call_impl(*args, **kwargs) 2024-08-20T22:11:12.7064663Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:11:12.7065505Z return forward_call(*args, **kwargs) 2024-08-20T22:11:12.7066452Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:11:12.7067295Z return self._torchdynamo_orig_callable( 2024-08-20T22:11:12.7068245Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:11:12.7069050Z return _compile( 2024-08-20T22:11:12.7069882Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:11:12.7070926Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:11:12.7071989Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:11:12.7072907Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:11:12.7074050Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:11:12.7074857Z return function(*args, **kwargs) 2024-08-20T22:11:12.7075793Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:11:12.7076695Z out_code = transform_code_object(code, transform) 2024-08-20T22:11:12.7077826Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:11:12.7078795Z transformations(instructions, code_options) 2024-08-20T22:11:12.7079717Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:11:12.7080488Z return fn(*args, **kwargs) 2024-08-20T22:11:12.7081350Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:11:12.7082123Z tracer.run() 2024-08-20T22:11:12.7082918Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:11:12.7083692Z super().run() 2024-08-20T22:11:12.7084569Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:11:12.7085353Z while self.step(): 2024-08-20T22:11:12.7086154Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:11:12.7087003Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:11:12.7087960Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:11:12.7088771Z return inner_fn(self, inst) 2024-08-20T22:11:12.7089686Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:11:12.7090539Z self.call_function(fn, args, {}) 2024-08-20T22:11:12.7091489Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:11:12.7092591Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:11:12.7093735Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:11:12.7094646Z return super().call_function(tx, args, kwargs) 2024-08-20T22:11:12.7095668Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:11:12.7096575Z return super().call_function(tx, args, kwargs) 2024-08-20T22:11:12.7097577Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:11:12.7098705Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:11:12.7100159Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:11:12.7101319Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:11:12.7102511Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:11:12.7103434Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:11:12.7104487Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:11:12.7105333Z tracer.run() 2024-08-20T22:11:12.7106143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:11:12.7106929Z while self.step(): 2024-08-20T22:11:12.7107772Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:11:12.7108637Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:11:12.7109696Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:11:12.7110521Z return inner_fn(self, inst) 2024-08-20T22:11:12.7111571Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:11:12.7112429Z self.call_function(fn, args, {}) 2024-08-20T22:11:12.7113374Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:11:12.7114455Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:11:12.7115611Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:11:12.7116523Z return super().call_function(tx, args, kwargs) 2024-08-20T22:11:12.7117548Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:11:12.7118459Z return super().call_function(tx, args, kwargs) 2024-08-20T22:11:12.7119525Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:11:12.7120678Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:11:12.7121863Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:11:12.7122931Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:11:12.7124009Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:11:12.7124878Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:11:12.7125851Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:11:12.7126632Z tracer.run() 2024-08-20T22:11:12.7127379Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:11:12.7128106Z while self.step(): 2024-08-20T22:11:12.7128881Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:11:12.7129681Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:11:12.7130615Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1739, in STORE_ATTR 2024-08-20T22:11:12.7131372Z assert ( 2024-08-20T22:11:12.7131761Z AssertionError: Mutating module attribute mask during export. 2024-08-20T22:11:12.7132140Z 2024-08-20T22:11:12.7132249Z from user code: 2024-08-20T22:11:12.7133067Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 50, in forward 2024-08-20T22:11:12.7133974Z x = transformer.forward(x, mask) 2024-08-20T22:11:12.7134922Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 47, in forward 2024-08-20T22:11:12.7136081Z self.lambda_module.set_mask(mask) 2024-08-20T22:11:12.7137077Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 16, in set_mask 2024-08-20T22:11:12.7138044Z self.mask = mask 2024-08-20T22:11:12.7138244Z 2024-08-20T22:11:12.7138596Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:11:12.7139225Z 2024-08-20T22:11:12.7139516Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:11:12.7140079Z fail_to_run 2024-08-20T22:11:12.7140672Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:12.7141379Z TIMING: 2024-08-20T22:11:12.7142021Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:12 | FakeTensorMode.__torch_dispatch__:43 2024-08-20T22:11:12.7142916Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:11:15.6839505Z 2024-08-20T22:11:18.7059185Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:11:18.7059664Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:11:18.7069550Z cpu eval Background_Matting 2024-08-20T22:11:18.7080295Z pass_due_to_skip 2024-08-20T22:11:18.7083591Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:18.7084750Z TIMING: 2024-08-20T22:11:18.7085006Z STATS: call_* op count: 0 2024-08-20T22:11:18.7085537Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:11:21.7781049Z 2024-08-20T22:11:23.9143139Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:11:23.9143650Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:11:23.9148829Z cpu eval LearningToPaint 2024-08-20T22:11:24.0092576Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:24.0424079Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:24.0692279Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:51.5592122Z pass 2024-08-20T22:11:51.5594868Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:11:51.5597533Z TIMING: backend_compile:9e-05 entire_frame_compile:0.58587 total_wall_time:0.58587 2024-08-20T22:11:51.5599196Z STATS: call_* op count: 71 | FakeTensor.__torch_dispatch__:358 | FakeTensorMode.__torch_dispatch__:5348 | ProxyTorchDispatchMode.__torch_dispatch__:961 2024-08-20T22:11:51.5600731Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique) 2024-08-20T22:11:55.0421251Z 2024-08-20T22:11:56.3214564Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-08-20T22:11:56.3412597Z 2024-08-20T22:11:56.3412786Z 2024-08-20T22:11:56.4418212Z 0% 0.00/528M [00:00 2024-08-20T22:20:18.8982724Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:20:18.8983701Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:20:18.8984568Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:20:18.8985545Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:18.8986341Z return node.target(*args, **kwargs) 2024-08-20T22:20:18.8987391Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:20:18.8988423Z r = self.shape_env.evaluate_expr( 2024-08-20T22:20:18.8989379Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:20:18.8990253Z return retlog(event.run(self)) 2024-08-20T22:20:18.8991172Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:20:18.8992003Z return self.f(*args, **kwargs) 2024-08-20T22:20:18.8993025Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:20:18.8994334Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:20:18.8995774Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:20:18.8996742Z raise self._make_data_dependent_error( 2024-08-20T22:20:18.8997993Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:20:18.8999520Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:20:18.9000275Z 2024-08-20T22:20:18.9000535Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:20:18.9001537Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:18.9002311Z return node.target(*args, **kwargs) 2024-08-20T22:20:18.9002590Z 2024-08-20T22:20:18.9002771Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:20:18.9003539Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:20:18.9004489Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:18.9005675Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:20:18.9006471Z 2024-08-20T22:20:18.9006600Z User Stack (most recent call last): 2024-08-20T22:20:18.9007005Z (snipped, see stack below for prefix) 2024-08-20T22:20:18.9007967Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:18.9008831Z return self.inference(batched_inputs) 2024-08-20T22:20:18.9009802Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:18.9010694Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:18.9011822Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:18.9012728Z images = ImageList.from_tensors( 2024-08-20T22:20:18.9013698Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:18.9014739Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:18.9015158Z 2024-08-20T22:20:18.9015397Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:18.9015804Z 2024-08-20T22:20:18.9016076Z During handling of the above exception, another exception occurred: 2024-08-20T22:20:18.9016502Z 2024-08-20T22:20:18.9016644Z Traceback (most recent call last): 2024-08-20T22:20:18.9017331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:20:18.9018070Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:20:18.9019109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:20:18.9020065Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:20:18.9020996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:20:18.9021702Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:20:18.9022707Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:20:18.9023584Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:20:18.9024505Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:20:18.9025300Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:20:18.9026311Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:20:18.9027210Z return self._call_impl(*args, **kwargs) 2024-08-20T22:20:18.9028157Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:20:18.9028988Z return forward_call(*args, **kwargs) 2024-08-20T22:20:18.9029865Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:20:18.9030637Z return fn(*args, **kwargs) 2024-08-20T22:20:18.9031634Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:20:18.9032452Z return self._call_impl(*args, **kwargs) 2024-08-20T22:20:18.9033328Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:20:18.9034108Z return forward_call(*args, **kwargs) 2024-08-20T22:20:18.9035158Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:20:18.9035985Z return self._torchdynamo_orig_callable( 2024-08-20T22:20:18.9036898Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:20:18.9037677Z return _compile( 2024-08-20T22:20:18.9038482Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:20:18.9039406Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:20:18.9040452Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:20:18.9041367Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:20:18.9042364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:20:18.9043176Z return function(*args, **kwargs) 2024-08-20T22:20:18.9044153Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:20:18.9045047Z out_code = transform_code_object(code, transform) 2024-08-20T22:20:18.9046169Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:20:18.9047227Z transformations(instructions, code_options) 2024-08-20T22:20:18.9048081Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:20:18.9048814Z return fn(*args, **kwargs) 2024-08-20T22:20:18.9049634Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:20:18.9050377Z tracer.run() 2024-08-20T22:20:18.9051114Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:20:18.9051844Z super().run() 2024-08-20T22:20:18.9052586Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:18.9053320Z while self.step(): 2024-08-20T22:20:18.9054139Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:18.9055286Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:18.9056274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:18.9057098Z return inner_fn(self, inst) 2024-08-20T22:20:18.9058032Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:20:18.9058983Z self.call_function(fn, args, {}) 2024-08-20T22:20:18.9059953Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:18.9061070Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:18.9062266Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:20:18.9063187Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:18.9064232Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:18.9065161Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:18.9066200Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:18.9067269Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:18.9068538Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:18.9069702Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:18.9071071Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:18.9071940Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:18.9072890Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:18.9073679Z tracer.run() 2024-08-20T22:20:18.9074429Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:18.9075164Z while self.step(): 2024-08-20T22:20:18.9075923Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:18.9076719Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:18.9077627Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:18.9078390Z return inner_fn(self, inst) 2024-08-20T22:20:18.9079375Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:20:18.9080194Z self.call_function(fn, args, {}) 2024-08-20T22:20:18.9081089Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:18.9082123Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:18.9083201Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:20:18.9084064Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:18.9085028Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:18.9085879Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:18.9086845Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:18.9088047Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:18.9089364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:18.9090487Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:18.9091640Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:18.9092539Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:18.9093562Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:18.9094382Z tracer.run() 2024-08-20T22:20:18.9095162Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:18.9095928Z while self.step(): 2024-08-20T22:20:18.9096735Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:18.9097570Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:18.9098749Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:18.9099584Z return inner_fn(self, inst) 2024-08-20T22:20:18.9100542Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:20:18.9101444Z self.call_function(fn, args, kwargs) 2024-08-20T22:20:18.9102423Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:18.9103546Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:18.9104724Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:18.9105661Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:18.9106714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:18.9107803Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:18.9109064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:18.9110223Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:18.9111467Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:18.9112328Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:18.9113364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:18.9114131Z tracer.run() 2024-08-20T22:20:18.9114886Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:18.9115625Z while self.step(): 2024-08-20T22:20:18.9116401Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:18.9117184Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:18.9118090Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:18.9118853Z return inner_fn(self, inst) 2024-08-20T22:20:18.9119671Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:20:18.9120525Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:20:18.9121546Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:20:18.9122364Z return handler(tx, args, kwargs) 2024-08-20T22:20:18.9123404Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:20:18.9124274Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:20:18.9125183Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:20:18.9126104Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:20:18.9127158Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:20:18.9128150Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:20:18.9129155Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:20:18.9129911Z raise UserError( # noqa: B904 2024-08-20T22:20:18.9132416Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:20:18.9134563Z 2024-08-20T22:20:18.9134812Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:20:18.9135763Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:18.9136505Z return node.target(*args, **kwargs) 2024-08-20T22:20:18.9136778Z 2024-08-20T22:20:18.9136952Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:20:18.9137688Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:20:18.9138828Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:18.9140046Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:20:18.9140871Z 2024-08-20T22:20:18.9141002Z User Stack (most recent call last): 2024-08-20T22:20:18.9141420Z (snipped, see stack below for prefix) 2024-08-20T22:20:18.9142424Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:18.9143317Z return self.inference(batched_inputs) 2024-08-20T22:20:18.9144313Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:18.9145248Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:18.9146411Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:18.9147343Z images = ImageList.from_tensors( 2024-08-20T22:20:18.9148345Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:18.9149413Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:18.9149833Z 2024-08-20T22:20:18.9150078Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:18.9151341Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:20:18.9152088Z 2024-08-20T22:20:18.9152199Z from user code: 2024-08-20T22:20:18.9152994Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:18.9154028Z return self.inference(batched_inputs) 2024-08-20T22:20:18.9155406Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:18.9156467Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:18.9157568Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:18.9158486Z images = ImageList.from_tensors( 2024-08-20T22:20:18.9159476Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:18.9160562Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:18.9160975Z 2024-08-20T22:20:18.9161276Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:20:18.9161721Z 2024-08-20T22:20:18.9162009Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:20:18.9162566Z fail_to_run 2024-08-20T22:20:18.9163172Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:20:18.9163869Z TIMING: 2024-08-20T22:20:18.9164423Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:20:18.9165296Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:20:22.0646369Z 2024-08-20T22:20:34.5464165Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:20:34.5464653Z loading model: 0it [00:12, ?it/s] 2024-08-20T22:20:34.5484387Z cpu eval detectron2_fasterrcnn_r_101_dc5 2024-08-20T22:20:52.6981316Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_101_dc5. Setting accuracy check to cosine 2024-08-20T22:20:52.7113969Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:05.4888291Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:18.2342138Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:25.2602760Z W0820 22:21:25.259352 15811 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:21:25.2605131Z E0820 22:21:25.259523 15811 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:21:25.2860309Z ERROR:common: 2024-08-20T22:21:25.2860640Z Traceback (most recent call last): 2024-08-20T22:21:25.2864355Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:21:25.2865197Z return node.target(*args, **kwargs) 2024-08-20T22:21:25.2866277Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:21:25.2867423Z r = self.shape_env.evaluate_expr( 2024-08-20T22:21:25.2868408Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:21:25.2869298Z return retlog(event.run(self)) 2024-08-20T22:21:25.2870220Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:21:25.2871063Z return self.f(*args, **kwargs) 2024-08-20T22:21:25.2872079Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:21:25.2873393Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:21:25.2874851Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:21:25.2875828Z raise self._make_data_dependent_error( 2024-08-20T22:21:25.2877189Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:21:25.2878338Z 2024-08-20T22:21:25.2878609Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:21:25.2879645Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:21:25.2880444Z return node.target(*args, **kwargs) 2024-08-20T22:21:25.2880732Z 2024-08-20T22:21:25.2880937Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:21:25.2881709Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:21:25.2882673Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.2883887Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:21:25.2884707Z 2024-08-20T22:21:25.2884858Z User Stack (most recent call last): 2024-08-20T22:21:25.2885278Z (snipped, see stack below for prefix) 2024-08-20T22:21:25.2886253Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:21:25.2887136Z return self.inference(batched_inputs) 2024-08-20T22:21:25.2888136Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:21:25.2889063Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:21:25.2890144Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:21:25.2891070Z images = ImageList.from_tensors( 2024-08-20T22:21:25.2892056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:21:25.2893204Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:21:25.2893630Z 2024-08-20T22:21:25.2893874Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.2894280Z 2024-08-20T22:21:25.2894579Z The above exception was the direct cause of the following exception: 2024-08-20T22:21:25.2895023Z 2024-08-20T22:21:25.2895167Z Traceback (most recent call last): 2024-08-20T22:21:25.2896055Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:21:25.2896871Z ret_val = wrap_fake_exception( 2024-08-20T22:21:25.2897796Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:21:25.2898709Z return fn() 2024-08-20T22:21:25.2899468Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:21:25.2900359Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:21:25.2901427Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:21:25.2902320Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:21:25.2903287Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:21:25.2904078Z return node.target(*args, **kwargs) 2024-08-20T22:21:25.2905126Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:21:25.2906063Z r = self.shape_env.evaluate_expr( 2024-08-20T22:21:25.2907024Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:21:25.2907874Z return retlog(event.run(self)) 2024-08-20T22:21:25.2908788Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:21:25.2909632Z return self.f(*args, **kwargs) 2024-08-20T22:21:25.2910668Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:21:25.2912121Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:21:25.2913523Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:21:25.2914458Z raise self._make_data_dependent_error( 2024-08-20T22:21:25.2915683Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:21:25.2917167Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:21:25.2917782Z 2024-08-20T22:21:25.2918045Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:21:25.2919049Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:21:25.2919830Z return node.target(*args, **kwargs) 2024-08-20T22:21:25.2920109Z 2024-08-20T22:21:25.2920304Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:21:25.2921238Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:21:25.2922302Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.2923484Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:21:25.2924467Z 2024-08-20T22:21:25.2924613Z User Stack (most recent call last): 2024-08-20T22:21:25.2925019Z (snipped, see stack below for prefix) 2024-08-20T22:21:25.2926013Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:21:25.2926908Z return self.inference(batched_inputs) 2024-08-20T22:21:25.2927911Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:21:25.2928848Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:21:25.2929924Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:21:25.2930850Z images = ImageList.from_tensors( 2024-08-20T22:21:25.2931837Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:21:25.2933038Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:21:25.2933436Z 2024-08-20T22:21:25.2933677Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.2934082Z 2024-08-20T22:21:25.2934419Z During handling of the above exception, another exception occurred: 2024-08-20T22:21:25.2934867Z 2024-08-20T22:21:25.2934997Z Traceback (most recent call last): 2024-08-20T22:21:25.2935706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:21:25.2936453Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:21:25.2937219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:21:25.2938146Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:21:25.2939236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:21:25.2939962Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:21:25.2940968Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:21:25.2941862Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:21:25.2942799Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:21:25.2943621Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:21:25.2944688Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:21:25.2945598Z return self._call_impl(*args, **kwargs) 2024-08-20T22:21:25.2946553Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:21:25.2947397Z return forward_call(*args, **kwargs) 2024-08-20T22:21:25.2948261Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:21:25.2949036Z return fn(*args, **kwargs) 2024-08-20T22:21:25.2949978Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:21:25.2951004Z return self._call_impl(*args, **kwargs) 2024-08-20T22:21:25.2951926Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:21:25.2952746Z return forward_call(*args, **kwargs) 2024-08-20T22:21:25.2953654Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:21:25.2954839Z return self._torchdynamo_orig_callable( 2024-08-20T22:21:25.2955806Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:21:25.2956613Z return _compile( 2024-08-20T22:21:25.2957450Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:21:25.2958405Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:21:25.2959502Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:21:25.2960441Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:21:25.2961466Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:21:25.2962302Z return function(*args, **kwargs) 2024-08-20T22:21:25.2963255Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:21:25.2964158Z out_code = transform_code_object(code, transform) 2024-08-20T22:21:25.2965304Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:21:25.2966312Z transformations(instructions, code_options) 2024-08-20T22:21:25.2967444Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:21:25.2968164Z return fn(*args, **kwargs) 2024-08-20T22:21:25.2969148Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:21:25.2969900Z tracer.run() 2024-08-20T22:21:25.2970659Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:21:25.2971389Z super().run() 2024-08-20T22:21:25.2972143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:21:25.2972879Z while self.step(): 2024-08-20T22:21:25.2973656Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:21:25.2974442Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:21:25.2975346Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:21:25.2976109Z return inner_fn(self, inst) 2024-08-20T22:21:25.2976992Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:21:25.2977804Z self.call_function(fn, args, {}) 2024-08-20T22:21:25.2978940Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:21:25.2980193Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:21:25.2981380Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:21:25.2982320Z return super().call_function(tx, args, kwargs) 2024-08-20T22:21:25.2983361Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:21:25.2984297Z return super().call_function(tx, args, kwargs) 2024-08-20T22:21:25.2985337Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:21:25.2986429Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:21:25.2987689Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:21:25.2988850Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:21:25.2990035Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:21:25.2991066Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:21:25.2992037Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:21:25.2992805Z tracer.run() 2024-08-20T22:21:25.2993550Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:21:25.2994289Z while self.step(): 2024-08-20T22:21:25.2995071Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:21:25.2995861Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:21:25.2996767Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:21:25.2997523Z return inner_fn(self, inst) 2024-08-20T22:21:25.2998399Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:21:25.2999393Z self.call_function(fn, args, {}) 2024-08-20T22:21:25.3000329Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:21:25.3001413Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:21:25.3002560Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:21:25.3003529Z return super().call_function(tx, args, kwargs) 2024-08-20T22:21:25.3004557Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:21:25.3005469Z return super().call_function(tx, args, kwargs) 2024-08-20T22:21:25.3006490Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:21:25.3022892Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:21:25.3024381Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:21:25.3025541Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:21:25.3026743Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:21:25.3027681Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:21:25.3028730Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:21:25.3029759Z tracer.run() 2024-08-20T22:21:25.3030558Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:21:25.3031451Z while self.step(): 2024-08-20T22:21:25.3032269Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:21:25.3033111Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:21:25.3034060Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:21:25.3034863Z return inner_fn(self, inst) 2024-08-20T22:21:25.3035798Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:21:25.3036688Z self.call_function(fn, args, kwargs) 2024-08-20T22:21:25.3037714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:21:25.3038757Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:21:25.3039859Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:21:25.3040718Z return super().call_function(tx, args, kwargs) 2024-08-20T22:21:25.3041675Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:21:25.3042676Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:21:25.3043850Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:21:25.3044920Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:21:25.3046016Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:21:25.3046872Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:21:25.3047836Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:21:25.3048617Z tracer.run() 2024-08-20T22:21:25.3049363Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:21:25.3050090Z while self.step(): 2024-08-20T22:21:25.3050857Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:21:25.3051654Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:21:25.3052625Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:21:25.3053379Z return inner_fn(self, inst) 2024-08-20T22:21:25.3054382Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:21:25.3055482Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:21:25.3056584Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:21:25.3057462Z return handler(tx, args, kwargs) 2024-08-20T22:21:25.3058589Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:21:25.3059547Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:21:25.3060525Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:21:25.3061528Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:21:25.3062673Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:21:25.3063881Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:21:25.3064979Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:21:25.3065803Z raise UserError( # noqa: B904 2024-08-20T22:21:25.3068496Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:21:25.3070997Z 2024-08-20T22:21:25.3071267Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:21:25.3072209Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:21:25.3072949Z return node.target(*args, **kwargs) 2024-08-20T22:21:25.3073227Z 2024-08-20T22:21:25.3073400Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:21:25.3074127Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:21:25.3075008Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.3076136Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:21:25.3076904Z 2024-08-20T22:21:25.3077026Z User Stack (most recent call last): 2024-08-20T22:21:25.3077418Z (snipped, see stack below for prefix) 2024-08-20T22:21:25.3078341Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:21:25.3079150Z return self.inference(batched_inputs) 2024-08-20T22:21:25.3080073Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:21:25.3080938Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:21:25.3081945Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:21:25.3082797Z images = ImageList.from_tensors( 2024-08-20T22:21:25.3083711Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:21:25.3084708Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:21:25.3085087Z 2024-08-20T22:21:25.3085327Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:21:25.3086504Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:21:25.3087253Z 2024-08-20T22:21:25.3087349Z from user code: 2024-08-20T22:21:25.3088167Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:21:25.3088990Z return self.inference(batched_inputs) 2024-08-20T22:21:25.3089920Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:21:25.3090771Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:21:25.3091779Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:21:25.3092633Z images = ImageList.from_tensors( 2024-08-20T22:21:25.3093554Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:21:25.3094550Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:21:25.3094937Z 2024-08-20T22:21:25.3095203Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:21:25.3095697Z 2024-08-20T22:21:25.3095967Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:21:25.3096477Z fail_to_run 2024-08-20T22:21:25.3097034Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:25.3097672Z TIMING: 2024-08-20T22:21:25.3098266Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:21:25.3099286Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:21:28.5971337Z 2024-08-20T22:21:36.9322706Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:21:36.9323213Z loading model: 0it [00:08, ?it/s] 2024-08-20T22:21:36.9340629Z cpu eval detectron2_fasterrcnn_r_101_fpn 2024-08-20T22:21:53.7834791Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_101_fpn. Setting accuracy check to cosine 2024-08-20T22:21:53.8094621Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:01.2111466Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:08.6050053Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:12.8255590Z W0820 22:22:12.824811 16138 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:22:12.8257870Z E0820 22:22:12.824970 16138 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:22:12.8525120Z ERROR:common: 2024-08-20T22:22:12.8525489Z Traceback (most recent call last): 2024-08-20T22:22:12.8526498Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:12.8527327Z return node.target(*args, **kwargs) 2024-08-20T22:22:12.8528383Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:22:12.8529338Z r = self.shape_env.evaluate_expr( 2024-08-20T22:22:12.8530308Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:22:12.8531172Z return retlog(event.run(self)) 2024-08-20T22:22:12.8532091Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:22:12.8532914Z return self.f(*args, **kwargs) 2024-08-20T22:22:12.8533949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:22:12.8535474Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:22:12.8536971Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:22:12.8537955Z raise self._make_data_dependent_error( 2024-08-20T22:22:12.8539413Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:12.8540462Z 2024-08-20T22:22:12.8540731Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:12.8541773Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:12.8542571Z return node.target(*args, **kwargs) 2024-08-20T22:22:12.8542858Z 2024-08-20T22:22:12.8543058Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:12.8543844Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:12.8544818Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8546160Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:12.8546989Z 2024-08-20T22:22:12.8547137Z User Stack (most recent call last): 2024-08-20T22:22:12.8547548Z (snipped, see stack below for prefix) 2024-08-20T22:22:12.8548548Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:12.8549435Z return self.inference(batched_inputs) 2024-08-20T22:22:12.8550439Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:12.8551341Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:12.8552441Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:12.8553362Z images = ImageList.from_tensors( 2024-08-20T22:22:12.8554520Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:12.8555615Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:12.8556025Z 2024-08-20T22:22:12.8556272Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8556694Z 2024-08-20T22:22:12.8556981Z The above exception was the direct cause of the following exception: 2024-08-20T22:22:12.8557441Z 2024-08-20T22:22:12.8557573Z Traceback (most recent call last): 2024-08-20T22:22:12.8558475Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:22:12.8559279Z ret_val = wrap_fake_exception( 2024-08-20T22:22:12.8560209Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:22:12.8561027Z return fn() 2024-08-20T22:22:12.8561799Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:22:12.8562673Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:22:12.8563653Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:22:12.8564534Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:22:12.8565517Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:12.8566302Z return node.target(*args, **kwargs) 2024-08-20T22:22:12.8567432Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:22:12.8568351Z r = self.shape_env.evaluate_expr( 2024-08-20T22:22:12.8569379Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:22:12.8570231Z return retlog(event.run(self)) 2024-08-20T22:22:12.8571111Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:22:12.8571997Z return self.f(*args, **kwargs) 2024-08-20T22:22:12.8572989Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:22:12.8574266Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:22:12.8575656Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:22:12.8576593Z raise self._make_data_dependent_error( 2024-08-20T22:22:12.8577825Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:22:12.8579644Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:12.8580293Z 2024-08-20T22:22:12.8580559Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:12.8581587Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:12.8582391Z return node.target(*args, **kwargs) 2024-08-20T22:22:12.8582677Z 2024-08-20T22:22:12.8582867Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:12.8583658Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:12.8584627Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8585847Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:12.8586663Z 2024-08-20T22:22:12.8586816Z User Stack (most recent call last): 2024-08-20T22:22:12.8587226Z (snipped, see stack below for prefix) 2024-08-20T22:22:12.8588213Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:12.8589096Z return self.inference(batched_inputs) 2024-08-20T22:22:12.8590094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:12.8591120Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:12.8592185Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:12.8593083Z images = ImageList.from_tensors( 2024-08-20T22:22:12.8594053Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:12.8595095Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:12.8595509Z 2024-08-20T22:22:12.8595749Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8596153Z 2024-08-20T22:22:12.8596427Z During handling of the above exception, another exception occurred: 2024-08-20T22:22:12.8596859Z 2024-08-20T22:22:12.8596998Z Traceback (most recent call last): 2024-08-20T22:22:12.8597699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:22:12.8598428Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:22:12.8599200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:22:12.8600219Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:22:12.8601077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:22:12.8601736Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:22:12.8602666Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:22:12.8603487Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:22:12.8604349Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:22:12.8605089Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:22:12.8606193Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:22:12.8607070Z return self._call_impl(*args, **kwargs) 2024-08-20T22:22:12.8607999Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:22:12.8608806Z return forward_call(*args, **kwargs) 2024-08-20T22:22:12.8609664Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:22:12.8610413Z return fn(*args, **kwargs) 2024-08-20T22:22:12.8611399Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:22:12.8612279Z return self._call_impl(*args, **kwargs) 2024-08-20T22:22:12.8613190Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:22:12.8614002Z return forward_call(*args, **kwargs) 2024-08-20T22:22:12.8614911Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:22:12.8615736Z return self._torchdynamo_orig_callable( 2024-08-20T22:22:12.8616642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:22:12.8617419Z return _compile( 2024-08-20T22:22:12.8618229Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:22:12.8619412Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:22:12.8620500Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:22:12.8621448Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:22:12.8622474Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:22:12.8623304Z return function(*args, **kwargs) 2024-08-20T22:22:12.8624245Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:22:12.8625166Z out_code = transform_code_object(code, transform) 2024-08-20T22:22:12.8626321Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:22:12.8627327Z transformations(instructions, code_options) 2024-08-20T22:22:12.8628274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:22:12.8629062Z return fn(*args, **kwargs) 2024-08-20T22:22:12.8629943Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:22:12.8630750Z tracer.run() 2024-08-20T22:22:12.8631627Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:22:12.8632352Z super().run() 2024-08-20T22:22:12.8633097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:12.8633833Z while self.step(): 2024-08-20T22:22:12.8634606Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:12.8635657Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:12.8636618Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:12.8637430Z return inner_fn(self, inst) 2024-08-20T22:22:12.8638352Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:22:12.8639195Z self.call_function(fn, args, {}) 2024-08-20T22:22:12.8640136Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:12.8641238Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:12.8642389Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:22:12.8643301Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:12.8644312Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:12.8645215Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:12.8646290Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:12.8647346Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:12.8648573Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:12.8649693Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:12.8650840Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:12.8651752Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:12.8652765Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:12.8653580Z tracer.run() 2024-08-20T22:22:12.8654503Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:12.8655462Z while self.step(): 2024-08-20T22:22:12.8656292Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:12.8657161Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:12.8658145Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:12.8659036Z return inner_fn(self, inst) 2024-08-20T22:22:12.8659992Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:22:12.8660855Z self.call_function(fn, args, {}) 2024-08-20T22:22:12.8661832Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:12.8662952Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:12.8664128Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:22:12.8665049Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:12.8666100Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:12.8667026Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:12.8668063Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:12.8669134Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:12.8670519Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:12.8671750Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:12.8672912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:12.8673818Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:12.8674828Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:12.8675654Z tracer.run() 2024-08-20T22:22:12.8676437Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:12.8677215Z while self.step(): 2024-08-20T22:22:12.8678020Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:12.8678855Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:12.8679905Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:12.8680745Z return inner_fn(self, inst) 2024-08-20T22:22:12.8681631Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:22:12.8682472Z self.call_function(fn, args, kwargs) 2024-08-20T22:22:12.8683378Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:12.8684417Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:12.8685516Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:12.8686366Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:12.8687335Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:12.8688343Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:12.8689521Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:12.8690571Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:12.8691666Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:12.8692526Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:12.8693495Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:12.8694258Z tracer.run() 2024-08-20T22:22:12.8695003Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:12.8695741Z while self.step(): 2024-08-20T22:22:12.8696515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:12.8697317Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:12.8698212Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:12.8699232Z return inner_fn(self, inst) 2024-08-20T22:22:12.8700118Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:22:12.8701041Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:22:12.8702127Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:22:12.8703004Z return handler(tx, args, kwargs) 2024-08-20T22:22:12.8704131Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:22:12.8705085Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:22:12.8706052Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:22:12.8707050Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:22:12.8708195Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:22:12.8709273Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:22:12.8710348Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:22:12.8711170Z raise UserError( # noqa: B904 2024-08-20T22:22:12.8713889Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:12.8716233Z 2024-08-20T22:22:12.8716502Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:12.8717644Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:12.8718421Z return node.target(*args, **kwargs) 2024-08-20T22:22:12.8718700Z 2024-08-20T22:22:12.8718883Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:12.8719647Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:12.8720594Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8721782Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:12.8722579Z 2024-08-20T22:22:12.8722721Z User Stack (most recent call last): 2024-08-20T22:22:12.8723121Z (snipped, see stack below for prefix) 2024-08-20T22:22:12.8724082Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:12.8724946Z return self.inference(batched_inputs) 2024-08-20T22:22:12.8725924Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:12.8726812Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:12.8727876Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:12.8728780Z images = ImageList.from_tensors( 2024-08-20T22:22:12.8729753Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:12.8730799Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:12.8731303Z 2024-08-20T22:22:12.8731529Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:12.8732619Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:22:12.8733365Z 2024-08-20T22:22:12.8733475Z from user code: 2024-08-20T22:22:12.8734285Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:12.8735097Z return self.inference(batched_inputs) 2024-08-20T22:22:12.8736027Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:12.8736887Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:12.8737958Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:12.8739053Z images = ImageList.from_tensors( 2024-08-20T22:22:12.8740058Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:12.8741138Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:12.8741545Z 2024-08-20T22:22:12.8741847Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:22:12.8742291Z 2024-08-20T22:22:12.8742592Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:22:12.8743131Z fail_to_run 2024-08-20T22:22:12.8743731Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:12.8744432Z TIMING: 2024-08-20T22:22:12.8744995Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:43 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:22:12.8745864Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:22:16.0628722Z 2024-08-20T22:22:30.2302786Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:22:30.2304237Z loading model: 0it [00:14, ?it/s] 2024-08-20T22:22:30.2315980Z cpu eval detectron2_fasterrcnn_r_50_c4 2024-08-20T22:22:38.5891295Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_c4. Setting accuracy check to cosine 2024-08-20T22:22:38.6055495Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:01.6737215Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:25.0785024Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:36.9090707Z W0820 22:23:36.908223 16465 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:23:36.9092999Z E0820 22:23:36.908383 16465 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:23:36.9344832Z ERROR:common: 2024-08-20T22:23:36.9345459Z Traceback (most recent call last): 2024-08-20T22:23:36.9346478Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:36.9348055Z return node.target(*args, **kwargs) 2024-08-20T22:23:36.9349297Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:23:36.9350256Z r = self.shape_env.evaluate_expr( 2024-08-20T22:23:36.9351229Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:23:36.9352097Z return retlog(event.run(self)) 2024-08-20T22:23:36.9353032Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:23:36.9353871Z return self.f(*args, **kwargs) 2024-08-20T22:23:36.9355197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:23:36.9356525Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:23:36.9357971Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:23:36.9358935Z raise self._make_data_dependent_error( 2024-08-20T22:23:36.9360280Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:36.9361311Z 2024-08-20T22:23:36.9361831Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:36.9362886Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:36.9363693Z return node.target(*args, **kwargs) 2024-08-20T22:23:36.9364075Z 2024-08-20T22:23:36.9364264Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:36.9365052Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:36.9366019Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9367236Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:36.9368060Z 2024-08-20T22:23:36.9368191Z User Stack (most recent call last): 2024-08-20T22:23:36.9368611Z (snipped, see stack below for prefix) 2024-08-20T22:23:36.9369616Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:36.9370505Z return self.inference(batched_inputs) 2024-08-20T22:23:36.9371491Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:36.9372560Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:36.9373666Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:36.9374595Z images = ImageList.from_tensors( 2024-08-20T22:23:36.9375576Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:36.9376660Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:36.9377080Z 2024-08-20T22:23:36.9377326Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9377729Z 2024-08-20T22:23:36.9378031Z The above exception was the direct cause of the following exception: 2024-08-20T22:23:36.9378575Z 2024-08-20T22:23:36.9378725Z Traceback (most recent call last): 2024-08-20T22:23:36.9379624Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:23:36.9380448Z ret_val = wrap_fake_exception( 2024-08-20T22:23:36.9381378Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:23:36.9382203Z return fn() 2024-08-20T22:23:36.9382958Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:23:36.9383838Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:23:36.9384815Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:23:36.9385689Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:23:36.9386657Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:36.9387441Z return node.target(*args, **kwargs) 2024-08-20T22:23:36.9388494Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:23:36.9389426Z r = self.shape_env.evaluate_expr( 2024-08-20T22:23:36.9390410Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:23:36.9391351Z return retlog(event.run(self)) 2024-08-20T22:23:36.9392243Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:23:36.9393052Z return self.f(*args, **kwargs) 2024-08-20T22:23:36.9394056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:23:36.9395396Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:23:36.9396796Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:23:36.9397746Z raise self._make_data_dependent_error( 2024-08-20T22:23:36.9398981Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:23:36.9400480Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:36.9401097Z 2024-08-20T22:23:36.9401358Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:36.9402356Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:36.9403132Z return node.target(*args, **kwargs) 2024-08-20T22:23:36.9403411Z 2024-08-20T22:23:36.9403611Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:36.9404366Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:36.9405382Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9406561Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:36.9407363Z 2024-08-20T22:23:36.9407504Z User Stack (most recent call last): 2024-08-20T22:23:36.9407917Z (snipped, see stack below for prefix) 2024-08-20T22:23:36.9408873Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:36.9409741Z return self.inference(batched_inputs) 2024-08-20T22:23:36.9410716Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:36.9411627Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:36.9412684Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:36.9413594Z images = ImageList.from_tensors( 2024-08-20T22:23:36.9414562Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:36.9415620Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:36.9416021Z 2024-08-20T22:23:36.9416262Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9416666Z 2024-08-20T22:23:36.9416943Z During handling of the above exception, another exception occurred: 2024-08-20T22:23:36.9417387Z 2024-08-20T22:23:36.9417516Z Traceback (most recent call last): 2024-08-20T22:23:36.9418215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:23:36.9419202Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:23:36.9419989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:23:36.9420955Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:23:36.9421819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:23:36.9422540Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:23:36.9423539Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:23:36.9424416Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:23:36.9425345Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:23:36.9426157Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:23:36.9427216Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:23:36.9428115Z return self._call_impl(*args, **kwargs) 2024-08-20T22:23:36.9429069Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:23:36.9429915Z return forward_call(*args, **kwargs) 2024-08-20T22:23:36.9430797Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:23:36.9431627Z return fn(*args, **kwargs) 2024-08-20T22:23:36.9432502Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:23:36.9433340Z return self._call_impl(*args, **kwargs) 2024-08-20T22:23:36.9434205Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:23:36.9434982Z return forward_call(*args, **kwargs) 2024-08-20T22:23:36.9435850Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:23:36.9436826Z return self._torchdynamo_orig_callable( 2024-08-20T22:23:36.9437796Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:23:36.9438570Z return _compile( 2024-08-20T22:23:36.9439377Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:23:36.9440298Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:23:36.9441360Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:23:36.9442270Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:23:36.9443265Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:23:36.9444074Z return function(*args, **kwargs) 2024-08-20T22:23:36.9445007Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:23:36.9445888Z out_code = transform_code_object(code, transform) 2024-08-20T22:23:36.9447011Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:23:36.9447989Z transformations(instructions, code_options) 2024-08-20T22:23:36.9448967Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:23:36.9449680Z return fn(*args, **kwargs) 2024-08-20T22:23:36.9450493Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:23:36.9451236Z tracer.run() 2024-08-20T22:23:36.9451980Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:23:36.9452707Z super().run() 2024-08-20T22:23:36.9453450Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:36.9454311Z while self.step(): 2024-08-20T22:23:36.9455310Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:36.9456162Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:36.9457145Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:36.9457976Z return inner_fn(self, inst) 2024-08-20T22:23:36.9459007Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:23:36.9459888Z self.call_function(fn, args, {}) 2024-08-20T22:23:36.9460844Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:36.9462117Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:36.9463319Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:23:36.9464263Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:36.9465295Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:36.9466232Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:36.9467274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:36.9468357Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:36.9469622Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:36.9470868Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:36.9471965Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:36.9472904Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:36.9473875Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:36.9474645Z tracer.run() 2024-08-20T22:23:36.9475404Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:36.9476146Z while self.step(): 2024-08-20T22:23:36.9476919Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:36.9477703Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:36.9478617Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:36.9479382Z return inner_fn(self, inst) 2024-08-20T22:23:36.9480436Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:23:36.9481283Z self.call_function(fn, args, {}) 2024-08-20T22:23:36.9482222Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:36.9483316Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:36.9484470Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:23:36.9485381Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:36.9486394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:36.9487303Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:36.9488324Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:36.9489575Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:36.9490843Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:36.9491995Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:36.9493175Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:36.9494113Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:36.9495149Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:36.9495989Z tracer.run() 2024-08-20T22:23:36.9496856Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:36.9497650Z while self.step(): 2024-08-20T22:23:36.9498547Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:36.9499428Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:36.9500416Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:36.9501249Z return inner_fn(self, inst) 2024-08-20T22:23:36.9502219Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:23:36.9503111Z self.call_function(fn, args, kwargs) 2024-08-20T22:23:36.9504084Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:36.9505205Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:36.9506393Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:36.9507378Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:36.9508429Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:36.9509520Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:36.9510812Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:36.9512044Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:36.9513177Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:36.9514081Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:36.9515097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:36.9515918Z tracer.run() 2024-08-20T22:23:36.9516688Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:36.9517464Z while self.step(): 2024-08-20T22:23:36.9518275Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:36.9519115Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:36.9520055Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:36.9520851Z return inner_fn(self, inst) 2024-08-20T22:23:36.9521706Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:23:36.9522614Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:23:36.9523670Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:23:36.9524530Z return handler(tx, args, kwargs) 2024-08-20T22:23:36.9525560Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:23:36.9526488Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:23:36.9527421Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:23:36.9528390Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:23:36.9529499Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:23:36.9530549Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:23:36.9531722Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:23:36.9532518Z raise UserError( # noqa: B904 2024-08-20T22:23:36.9535171Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:36.9537380Z 2024-08-20T22:23:36.9537641Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:36.9538883Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:36.9539689Z return node.target(*args, **kwargs) 2024-08-20T22:23:36.9539976Z 2024-08-20T22:23:36.9540184Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:36.9540959Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:36.9541992Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9543206Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:36.9544031Z 2024-08-20T22:23:36.9544176Z User Stack (most recent call last): 2024-08-20T22:23:36.9544581Z (snipped, see stack below for prefix) 2024-08-20T22:23:36.9545589Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:36.9546474Z return self.inference(batched_inputs) 2024-08-20T22:23:36.9547474Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:36.9548404Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:36.9549499Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:36.9550439Z images = ImageList.from_tensors( 2024-08-20T22:23:36.9551532Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:36.9552587Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:36.9552987Z 2024-08-20T22:23:36.9553225Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:36.9554668Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:23:36.9555496Z 2024-08-20T22:23:36.9555603Z from user code: 2024-08-20T22:23:36.9556486Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:36.9557370Z return self.inference(batched_inputs) 2024-08-20T22:23:36.9558375Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:36.9559306Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:36.9560391Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:36.9561317Z images = ImageList.from_tensors( 2024-08-20T22:23:36.9562292Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:36.9563368Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:36.9563777Z 2024-08-20T22:23:36.9564079Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:23:36.9564524Z 2024-08-20T22:23:36.9564824Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:23:36.9565476Z fail_to_run 2024-08-20T22:23:36.9566088Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:36.9566806Z TIMING: 2024-08-20T22:23:36.9567584Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:23:36.9568393Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:23:40.0241401Z 2024-08-20T22:23:51.6861981Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:23:51.6862464Z loading model: 0it [00:11, ?it/s] 2024-08-20T22:23:51.6873177Z cpu eval detectron2_fasterrcnn_r_50_dc5 2024-08-20T22:24:06.0602485Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_dc5. Setting accuracy check to cosine 2024-08-20T22:24:06.1103153Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:21.1102765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:36.0635780Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:44.3789902Z W0820 22:24:44.378466 16792 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:24:44.3792212Z E0820 22:24:44.378636 16792 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:24:44.4056490Z ERROR:common: 2024-08-20T22:24:44.4056839Z Traceback (most recent call last): 2024-08-20T22:24:44.4057814Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:44.4058698Z return node.target(*args, **kwargs) 2024-08-20T22:24:44.4059788Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:24:44.4060731Z r = self.shape_env.evaluate_expr( 2024-08-20T22:24:44.4061727Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:24:44.4062604Z return retlog(event.run(self)) 2024-08-20T22:24:44.4063511Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:24:44.4064343Z return self.f(*args, **kwargs) 2024-08-20T22:24:44.4065385Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:24:44.4066690Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:24:44.4068142Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:24:44.4069096Z raise self._make_data_dependent_error( 2024-08-20T22:24:44.4070445Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:44.4071485Z 2024-08-20T22:24:44.4071755Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:44.4072789Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:44.4073583Z return node.target(*args, **kwargs) 2024-08-20T22:24:44.4073870Z 2024-08-20T22:24:44.4074061Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:44.4074848Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:44.4075812Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4077317Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:44.4078135Z 2024-08-20T22:24:44.4078268Z User Stack (most recent call last): 2024-08-20T22:24:44.4078697Z (snipped, see stack below for prefix) 2024-08-20T22:24:44.4079702Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:44.4080597Z return self.inference(batched_inputs) 2024-08-20T22:24:44.4081584Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:44.4082514Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:44.4083608Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:44.4084529Z images = ImageList.from_tensors( 2024-08-20T22:24:44.4085513Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:44.4086604Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:44.4087139Z 2024-08-20T22:24:44.4087386Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4087876Z 2024-08-20T22:24:44.4088162Z The above exception was the direct cause of the following exception: 2024-08-20T22:24:44.4088620Z 2024-08-20T22:24:44.4088752Z Traceback (most recent call last): 2024-08-20T22:24:44.4089660Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:24:44.4090480Z ret_val = wrap_fake_exception( 2024-08-20T22:24:44.4091402Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:24:44.4092217Z return fn() 2024-08-20T22:24:44.4092983Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:24:44.4093871Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:24:44.4094852Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:24:44.4095724Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:24:44.4096706Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:44.4097495Z return node.target(*args, **kwargs) 2024-08-20T22:24:44.4098625Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:24:44.4099556Z r = self.shape_env.evaluate_expr( 2024-08-20T22:24:44.4100524Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:24:44.4101380Z return retlog(event.run(self)) 2024-08-20T22:24:44.4102305Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:24:44.4103120Z return self.f(*args, **kwargs) 2024-08-20T22:24:44.4104152Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:24:44.4105462Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:24:44.4106893Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:24:44.4107855Z raise self._make_data_dependent_error( 2024-08-20T22:24:44.4109108Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:24:44.4110633Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:44.4111355Z 2024-08-20T22:24:44.4111622Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:44.4112650Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:44.4113434Z return node.target(*args, **kwargs) 2024-08-20T22:24:44.4113732Z 2024-08-20T22:24:44.4113919Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:44.4114709Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:44.4115671Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4116876Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:44.4117699Z 2024-08-20T22:24:44.4117832Z User Stack (most recent call last): 2024-08-20T22:24:44.4118252Z (snipped, see stack below for prefix) 2024-08-20T22:24:44.4119246Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:44.4120237Z return self.inference(batched_inputs) 2024-08-20T22:24:44.4121224Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:44.4122153Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:44.4123248Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:44.4124172Z images = ImageList.from_tensors( 2024-08-20T22:24:44.4125150Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:44.4126234Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:44.4126653Z 2024-08-20T22:24:44.4126900Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4127304Z 2024-08-20T22:24:44.4127597Z During handling of the above exception, another exception occurred: 2024-08-20T22:24:44.4128043Z 2024-08-20T22:24:44.4128186Z Traceback (most recent call last): 2024-08-20T22:24:44.4128888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:24:44.4129653Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:24:44.4130437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:24:44.4131382Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:24:44.4132221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:24:44.4132936Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:24:44.4133937Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:24:44.4134823Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:24:44.4135734Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:24:44.4136549Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:24:44.4137559Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:24:44.4138540Z return self._call_impl(*args, **kwargs) 2024-08-20T22:24:44.4139481Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:24:44.4140321Z return forward_call(*args, **kwargs) 2024-08-20T22:24:44.4141198Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:24:44.4141975Z return fn(*args, **kwargs) 2024-08-20T22:24:44.4142906Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:24:44.4143882Z return self._call_impl(*args, **kwargs) 2024-08-20T22:24:44.4144837Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:24:44.4145681Z return forward_call(*args, **kwargs) 2024-08-20T22:24:44.4146613Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:24:44.4147449Z return self._torchdynamo_orig_callable( 2024-08-20T22:24:44.4148398Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:24:44.4149190Z return _compile( 2024-08-20T22:24:44.4150020Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:24:44.4150959Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:24:44.4152052Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:24:44.4152992Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:24:44.4154085Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:24:44.4155089Z return function(*args, **kwargs) 2024-08-20T22:24:44.4156051Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:24:44.4156965Z out_code = transform_code_object(code, transform) 2024-08-20T22:24:44.4158113Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:24:44.4159104Z transformations(instructions, code_options) 2024-08-20T22:24:44.4160038Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:24:44.4160824Z return fn(*args, **kwargs) 2024-08-20T22:24:44.4161708Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:24:44.4162514Z tracer.run() 2024-08-20T22:24:44.4163310Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:24:44.4164099Z super().run() 2024-08-20T22:24:44.4164903Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:44.4165697Z while self.step(): 2024-08-20T22:24:44.4166518Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:44.4167450Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:44.4168354Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:44.4169121Z return inner_fn(self, inst) 2024-08-20T22:24:44.4169993Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:24:44.4170801Z self.call_function(fn, args, {}) 2024-08-20T22:24:44.4171691Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:44.4172733Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:44.4173825Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:24:44.4174689Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:44.4175648Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:44.4176506Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:44.4192992Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:44.4194088Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:44.4195357Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:44.4196490Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:44.4197653Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:44.4198552Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:44.4199578Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:44.4200409Z tracer.run() 2024-08-20T22:24:44.4201197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:44.4201964Z while self.step(): 2024-08-20T22:24:44.4202779Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:44.4203741Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:44.4204707Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:44.4205500Z return inner_fn(self, inst) 2024-08-20T22:24:44.4206426Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:24:44.4207281Z self.call_function(fn, args, {}) 2024-08-20T22:24:44.4208224Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:44.4209311Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:44.4210454Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:24:44.4211449Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:44.4212425Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:44.4213286Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:44.4214242Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:44.4215253Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:44.4216433Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:44.4217505Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:44.4218868Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:44.4219811Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:44.4220872Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:44.4221717Z tracer.run() 2024-08-20T22:24:44.4222522Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:44.4223308Z while self.step(): 2024-08-20T22:24:44.4224141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:44.4225007Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:44.4225987Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:44.4226804Z return inner_fn(self, inst) 2024-08-20T22:24:44.4227839Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:24:44.4228744Z self.call_function(fn, args, kwargs) 2024-08-20T22:24:44.4229733Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:44.4230957Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:44.4232110Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:44.4233024Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:44.4234045Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:44.4235104Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:44.4236327Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:44.4237448Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:44.4238660Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:44.4239572Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:44.4240659Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:44.4241438Z tracer.run() 2024-08-20T22:24:44.4242186Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:44.4242928Z while self.step(): 2024-08-20T22:24:44.4243688Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:44.4244488Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:44.4245569Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:44.4246382Z return inner_fn(self, inst) 2024-08-20T22:24:44.4247230Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:24:44.4248133Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:24:44.4249208Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:24:44.4250058Z return handler(tx, args, kwargs) 2024-08-20T22:24:44.4251095Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:24:44.4252009Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:24:44.4252962Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:24:44.4253945Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:24:44.4255406Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:24:44.4256478Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:24:44.4257572Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:24:44.4258460Z raise UserError( # noqa: B904 2024-08-20T22:24:44.4261191Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:44.4263581Z 2024-08-20T22:24:44.4263864Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:44.4264902Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:44.4265694Z return node.target(*args, **kwargs) 2024-08-20T22:24:44.4265993Z 2024-08-20T22:24:44.4266183Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:44.4266978Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:44.4267949Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4269159Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:44.4269992Z 2024-08-20T22:24:44.4270235Z User Stack (most recent call last): 2024-08-20T22:24:44.4270740Z (snipped, see stack below for prefix) 2024-08-20T22:24:44.4271660Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:44.4272485Z return self.inference(batched_inputs) 2024-08-20T22:24:44.4273479Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:44.4274345Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:44.4275355Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:44.4276213Z images = ImageList.from_tensors( 2024-08-20T22:24:44.4277122Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:44.4278126Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:44.4278521Z 2024-08-20T22:24:44.4278749Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:44.4279852Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:24:44.4280603Z 2024-08-20T22:24:44.4280716Z from user code: 2024-08-20T22:24:44.4281510Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:44.4282335Z return self.inference(batched_inputs) 2024-08-20T22:24:44.4283267Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:44.4284132Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:44.4285135Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:44.4285998Z images = ImageList.from_tensors( 2024-08-20T22:24:44.4286922Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:44.4287919Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:44.4288305Z 2024-08-20T22:24:44.4288571Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:24:44.4288997Z 2024-08-20T22:24:44.4289266Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:24:44.4289781Z fail_to_run 2024-08-20T22:24:44.4290341Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:44.4290981Z TIMING: 2024-08-20T22:24:44.4291506Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:24:44.4292325Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:24:47.5355491Z 2024-08-20T22:24:54.3776437Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:24:54.3776908Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:24:54.3786206Z cpu eval detectron2_fasterrcnn_r_50_fpn 2024-08-20T22:25:08.5020593Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_fpn. Setting accuracy check to cosine 2024-08-20T22:25:08.5419564Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:16.8693537Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:25.1440622Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:29.7309261Z W0820 22:25:29.730419 17119 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:25:29.7311521Z E0820 22:25:29.730586 17119 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:25:29.7566110Z ERROR:common: 2024-08-20T22:25:29.7566736Z Traceback (most recent call last): 2024-08-20T22:25:29.7568213Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:25:29.7570731Z return node.target(*args, **kwargs) 2024-08-20T22:25:29.7571939Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:25:29.7572897Z r = self.shape_env.evaluate_expr( 2024-08-20T22:25:29.7573867Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:25:29.7574737Z return retlog(event.run(self)) 2024-08-20T22:25:29.7575678Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:25:29.7576523Z return self.f(*args, **kwargs) 2024-08-20T22:25:29.7577562Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:25:29.7578990Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:25:29.7580455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:25:29.7581457Z raise self._make_data_dependent_error( 2024-08-20T22:25:29.7582800Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:25:29.7583824Z 2024-08-20T22:25:29.7584111Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:25:29.7585145Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:25:29.7585931Z return node.target(*args, **kwargs) 2024-08-20T22:25:29.7586230Z 2024-08-20T22:25:29.7586427Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:25:29.7587210Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:25:29.7588185Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7589388Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:25:29.7590223Z 2024-08-20T22:25:29.7590355Z User Stack (most recent call last): 2024-08-20T22:25:29.7590775Z (snipped, see stack below for prefix) 2024-08-20T22:25:29.7591767Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:25:29.7592645Z return self.inference(batched_inputs) 2024-08-20T22:25:29.7593649Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:25:29.7594764Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:25:29.7595875Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:25:29.7596809Z images = ImageList.from_tensors( 2024-08-20T22:25:29.7597791Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:25:29.7598875Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:25:29.7599287Z 2024-08-20T22:25:29.7599551Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7599959Z 2024-08-20T22:25:29.7600260Z The above exception was the direct cause of the following exception: 2024-08-20T22:25:29.7600704Z 2024-08-20T22:25:29.7600835Z Traceback (most recent call last): 2024-08-20T22:25:29.7601741Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:25:29.7602561Z ret_val = wrap_fake_exception( 2024-08-20T22:25:29.7603488Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:25:29.7604370Z return fn() 2024-08-20T22:25:29.7605141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:25:29.7606029Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:25:29.7607022Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:25:29.7607905Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:25:29.7608873Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:25:29.7609660Z return node.target(*args, **kwargs) 2024-08-20T22:25:29.7610714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:25:29.7611659Z r = self.shape_env.evaluate_expr( 2024-08-20T22:25:29.7612611Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:25:29.7613490Z return retlog(event.run(self)) 2024-08-20T22:25:29.7614403Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:25:29.7615241Z return self.f(*args, **kwargs) 2024-08-20T22:25:29.7616260Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:25:29.7617576Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:25:29.7619088Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:25:29.7620061Z raise self._make_data_dependent_error( 2024-08-20T22:25:29.7621333Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:25:29.7622836Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:25:29.7623487Z 2024-08-20T22:25:29.7623753Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:25:29.7624783Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:25:29.7625582Z return node.target(*args, **kwargs) 2024-08-20T22:25:29.7625866Z 2024-08-20T22:25:29.7626054Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:25:29.7626847Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:25:29.7627899Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7629125Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:25:29.7629953Z 2024-08-20T22:25:29.7630101Z User Stack (most recent call last): 2024-08-20T22:25:29.7630512Z (snipped, see stack below for prefix) 2024-08-20T22:25:29.7631513Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:25:29.7632414Z return self.inference(batched_inputs) 2024-08-20T22:25:29.7633417Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:25:29.7634333Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:25:29.7635431Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:25:29.7636360Z images = ImageList.from_tensors( 2024-08-20T22:25:29.7637355Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:25:29.7638537Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:25:29.7638965Z 2024-08-20T22:25:29.7639214Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7639636Z 2024-08-20T22:25:29.7639921Z During handling of the above exception, another exception occurred: 2024-08-20T22:25:29.7640363Z 2024-08-20T22:25:29.7640509Z Traceback (most recent call last): 2024-08-20T22:25:29.7641327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:25:29.7642056Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:25:29.7642901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:25:29.7643815Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:25:29.7644657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:25:29.7645360Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:25:29.7646348Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:25:29.7647195Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:25:29.7648100Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:25:29.7648895Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:25:29.7649877Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:25:29.7650742Z return self._call_impl(*args, **kwargs) 2024-08-20T22:25:29.7651668Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:25:29.7652490Z return forward_call(*args, **kwargs) 2024-08-20T22:25:29.7653343Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:25:29.7654091Z return fn(*args, **kwargs) 2024-08-20T22:25:29.7655391Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:25:29.7656278Z return self._call_impl(*args, **kwargs) 2024-08-20T22:25:29.7657231Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:25:29.7658068Z return forward_call(*args, **kwargs) 2024-08-20T22:25:29.7659078Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:25:29.7659914Z return self._torchdynamo_orig_callable( 2024-08-20T22:25:29.7660857Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:25:29.7661798Z return _compile( 2024-08-20T22:25:29.7662639Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:25:29.7663579Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:25:29.7664670Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:25:29.7665614Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:25:29.7666642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:25:29.7667462Z return function(*args, **kwargs) 2024-08-20T22:25:29.7668418Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:25:29.7669333Z out_code = transform_code_object(code, transform) 2024-08-20T22:25:29.7670592Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:25:29.7671527Z transformations(instructions, code_options) 2024-08-20T22:25:29.7672473Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:25:29.7673198Z return fn(*args, **kwargs) 2024-08-20T22:25:29.7674015Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:25:29.7674767Z tracer.run() 2024-08-20T22:25:29.7675509Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:25:29.7676248Z super().run() 2024-08-20T22:25:29.7676996Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:25:29.7677730Z while self.step(): 2024-08-20T22:25:29.7678490Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:25:29.7679283Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:25:29.7680189Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:25:29.7680954Z return inner_fn(self, inst) 2024-08-20T22:25:29.7681819Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:25:29.7682630Z self.call_function(fn, args, {}) 2024-08-20T22:25:29.7683528Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:25:29.7684569Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:25:29.7685652Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:25:29.7686523Z return super().call_function(tx, args, kwargs) 2024-08-20T22:25:29.7687497Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:25:29.7688353Z return super().call_function(tx, args, kwargs) 2024-08-20T22:25:29.7689319Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:25:29.7690311Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:25:29.7691487Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:25:29.7692556Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:25:29.7693648Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:25:29.7694566Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:25:29.7695542Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:25:29.7696330Z tracer.run() 2024-08-20T22:25:29.7697076Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:25:29.7697803Z while self.step(): 2024-08-20T22:25:29.7698644Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:25:29.7699667Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:25:29.7700660Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:25:29.7701487Z return inner_fn(self, inst) 2024-08-20T22:25:29.7702421Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:25:29.7703296Z self.call_function(fn, args, {}) 2024-08-20T22:25:29.7704260Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:25:29.7705450Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:25:29.7706616Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:25:29.7707538Z return super().call_function(tx, args, kwargs) 2024-08-20T22:25:29.7708584Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:25:29.7709510Z return super().call_function(tx, args, kwargs) 2024-08-20T22:25:29.7710533Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:25:29.7711708Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:25:29.7712949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:25:29.7714077Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:25:29.7715233Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:25:29.7716126Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:25:29.7717144Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:25:29.7717967Z tracer.run() 2024-08-20T22:25:29.7718751Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:25:29.7719614Z while self.step(): 2024-08-20T22:25:29.7720393Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:25:29.7721190Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:25:29.7722097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:25:29.7722852Z return inner_fn(self, inst) 2024-08-20T22:25:29.7723746Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:25:29.7724579Z self.call_function(fn, args, kwargs) 2024-08-20T22:25:29.7725487Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:25:29.7726512Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:25:29.7727610Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:25:29.7728474Z return super().call_function(tx, args, kwargs) 2024-08-20T22:25:29.7730057Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:25:29.7731087Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:25:29.7732261Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:25:29.7733327Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:25:29.7734421Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:25:29.7735288Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:25:29.7736243Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:25:29.7737030Z tracer.run() 2024-08-20T22:25:29.7737780Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:25:29.7738599Z while self.step(): 2024-08-20T22:25:29.7739655Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:25:29.7740516Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:25:29.7741499Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:25:29.7742330Z return inner_fn(self, inst) 2024-08-20T22:25:29.7743197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:25:29.7744125Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:25:29.7745224Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:25:29.7746100Z return handler(tx, args, kwargs) 2024-08-20T22:25:29.7747166Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:25:29.7748112Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:25:29.7749086Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:25:29.7750092Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:25:29.7751334Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:25:29.7752321Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:25:29.7753334Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:25:29.7754094Z raise UserError( # noqa: B904 2024-08-20T22:25:29.7757129Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:25:29.7759469Z 2024-08-20T22:25:29.7759740Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:25:29.7760774Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:25:29.7761560Z return node.target(*args, **kwargs) 2024-08-20T22:25:29.7761861Z 2024-08-20T22:25:29.7762050Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:25:29.7762831Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:25:29.7763798Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7765142Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:25:29.7765976Z 2024-08-20T22:25:29.7766109Z User Stack (most recent call last): 2024-08-20T22:25:29.7766639Z (snipped, see stack below for prefix) 2024-08-20T22:25:29.7767682Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:25:29.7768676Z return self.inference(batched_inputs) 2024-08-20T22:25:29.7769642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:25:29.7770550Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:25:29.7771616Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:25:29.7772522Z images = ImageList.from_tensors( 2024-08-20T22:25:29.7773480Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:25:29.7774626Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:25:29.7775038Z 2024-08-20T22:25:29.7775279Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:25:29.7776430Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:25:29.7777214Z 2024-08-20T22:25:29.7777328Z from user code: 2024-08-20T22:25:29.7778165Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:25:29.7779304Z return self.inference(batched_inputs) 2024-08-20T22:25:29.7780318Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:25:29.7781256Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:25:29.7782333Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:25:29.7783276Z images = ImageList.from_tensors( 2024-08-20T22:25:29.7784272Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:25:29.7785367Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:25:29.7785775Z 2024-08-20T22:25:29.7786077Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:25:29.7786527Z 2024-08-20T22:25:29.7786815Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:25:29.7787363Z fail_to_run 2024-08-20T22:25:29.7787971Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:29.7788668Z TIMING: 2024-08-20T22:25:29.7789222Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:43 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:25:29.7790098Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:25:32.9451662Z 2024-08-20T22:25:48.5759861Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:25:48.5760360Z loading model: 0it [00:15, ?it/s] 2024-08-20T22:25:48.5778618Z cpu eval detectron2_maskrcnn_r_101_c4 2024-08-20T22:25:59.5458108Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_101_c4. Setting accuracy check to cosine 2024-08-20T22:25:59.5467125Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:19.3841613Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:39.3835947Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:49.7411204Z W0820 22:26:49.740190 17446 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:26:49.7413574Z E0820 22:26:49.740347 17446 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:26:49.7665910Z ERROR:common: 2024-08-20T22:26:49.7666262Z Traceback (most recent call last): 2024-08-20T22:26:49.7667231Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:49.7668044Z return node.target(*args, **kwargs) 2024-08-20T22:26:49.7669181Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:26:49.7670336Z r = self.shape_env.evaluate_expr( 2024-08-20T22:26:49.7671312Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:26:49.7672915Z return retlog(event.run(self)) 2024-08-20T22:26:49.7674040Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:26:49.7675096Z return self.f(*args, **kwargs) 2024-08-20T22:26:49.7676156Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:26:49.7677467Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:26:49.7678975Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:26:49.7679939Z raise self._make_data_dependent_error( 2024-08-20T22:26:49.7681291Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:49.7682332Z 2024-08-20T22:26:49.7682608Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:49.7683658Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:49.7684470Z return node.target(*args, **kwargs) 2024-08-20T22:26:49.7684758Z 2024-08-20T22:26:49.7684962Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:49.7685754Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:49.7686716Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7687939Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:49.7688782Z 2024-08-20T22:26:49.7688996Z User Stack (most recent call last): 2024-08-20T22:26:49.7689421Z (snipped, see stack below for prefix) 2024-08-20T22:26:49.7690419Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:49.7691306Z return self.inference(batched_inputs) 2024-08-20T22:26:49.7692307Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:49.7693246Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:49.7694342Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:49.7695271Z images = ImageList.from_tensors( 2024-08-20T22:26:49.7696267Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:49.7697344Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:49.7697769Z 2024-08-20T22:26:49.7698018Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7698529Z 2024-08-20T22:26:49.7698964Z The above exception was the direct cause of the following exception: 2024-08-20T22:26:49.7699415Z 2024-08-20T22:26:49.7699568Z Traceback (most recent call last): 2024-08-20T22:26:49.7700484Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:26:49.7701309Z ret_val = wrap_fake_exception( 2024-08-20T22:26:49.7702249Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:26:49.7703077Z return fn() 2024-08-20T22:26:49.7703834Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:26:49.7704721Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:26:49.7705704Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:26:49.7706593Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:26:49.7707574Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:49.7708454Z return node.target(*args, **kwargs) 2024-08-20T22:26:49.7709504Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:26:49.7710460Z r = self.shape_env.evaluate_expr( 2024-08-20T22:26:49.7711434Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:26:49.7712291Z return retlog(event.run(self)) 2024-08-20T22:26:49.7713213Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:26:49.7714050Z return self.f(*args, **kwargs) 2024-08-20T22:26:49.7715072Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:26:49.7716377Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:26:49.7717824Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:26:49.7718789Z raise self._make_data_dependent_error( 2024-08-20T22:26:49.7720059Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:26:49.7721592Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:49.7722228Z 2024-08-20T22:26:49.7722498Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:49.7723526Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:49.7724332Z return node.target(*args, **kwargs) 2024-08-20T22:26:49.7724618Z 2024-08-20T22:26:49.7724821Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:49.7725602Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:49.7726572Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7727793Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:49.7728614Z 2024-08-20T22:26:49.7728759Z User Stack (most recent call last): 2024-08-20T22:26:49.7729186Z (snipped, see stack below for prefix) 2024-08-20T22:26:49.7730163Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:49.7731052Z return self.inference(batched_inputs) 2024-08-20T22:26:49.7732117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:49.7733049Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:49.7734141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:49.7735063Z images = ImageList.from_tensors( 2024-08-20T22:26:49.7736051Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:49.7737137Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:49.7737549Z 2024-08-20T22:26:49.7737796Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7738212Z 2024-08-20T22:26:49.7738571Z During handling of the above exception, another exception occurred: 2024-08-20T22:26:49.7739034Z 2024-08-20T22:26:49.7739164Z Traceback (most recent call last): 2024-08-20T22:26:49.7739886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:26:49.7740651Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:26:49.7741498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:26:49.7742451Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:26:49.7743308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:26:49.7744031Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:26:49.7745040Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:26:49.7745929Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:26:49.7746858Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:26:49.7747671Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:26:49.7748673Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:26:49.7749576Z return self._call_impl(*args, **kwargs) 2024-08-20T22:26:49.7750543Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:26:49.7751481Z return forward_call(*args, **kwargs) 2024-08-20T22:26:49.7752331Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:26:49.7753080Z return fn(*args, **kwargs) 2024-08-20T22:26:49.7753998Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:26:49.7755259Z return self._call_impl(*args, **kwargs) 2024-08-20T22:26:49.7756209Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:26:49.7757054Z return forward_call(*args, **kwargs) 2024-08-20T22:26:49.7757993Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:26:49.7758845Z return self._torchdynamo_orig_callable( 2024-08-20T22:26:49.7759791Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:26:49.7760581Z return _compile( 2024-08-20T22:26:49.7761409Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:26:49.7762359Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:26:49.7763451Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:26:49.7764382Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:26:49.7765406Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:26:49.7766366Z return function(*args, **kwargs) 2024-08-20T22:26:49.7767436Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:26:49.7768410Z out_code = transform_code_object(code, transform) 2024-08-20T22:26:49.7769482Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:26:49.7770420Z transformations(instructions, code_options) 2024-08-20T22:26:49.7771293Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:26:49.7772012Z return fn(*args, **kwargs) 2024-08-20T22:26:49.7772833Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:26:49.7773581Z tracer.run() 2024-08-20T22:26:49.7774340Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:26:49.7775079Z super().run() 2024-08-20T22:26:49.7775814Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:49.7776639Z while self.step(): 2024-08-20T22:26:49.7777418Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:49.7778218Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:49.7779380Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:49.7780213Z return inner_fn(self, inst) 2024-08-20T22:26:49.7781167Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:26:49.7782052Z self.call_function(fn, args, {}) 2024-08-20T22:26:49.7783013Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:49.7784142Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:49.7785340Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:26:49.7786273Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:49.7787303Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:49.7788234Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:49.7789283Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:49.7790375Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:49.7791732Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:49.7792790Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:49.7793890Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:49.7794750Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:49.7795717Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:49.7796484Z tracer.run() 2024-08-20T22:26:49.7797226Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:49.7797966Z while self.step(): 2024-08-20T22:26:49.7798905Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:49.7799736Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:49.7800752Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:49.7801568Z return inner_fn(self, inst) 2024-08-20T22:26:49.7802500Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:26:49.7803347Z self.call_function(fn, args, {}) 2024-08-20T22:26:49.7804282Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:49.7805374Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:49.7806529Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:26:49.7807439Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:49.7808448Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:49.7809357Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:49.7810375Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:49.7811487Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:49.7812720Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:49.7813848Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:49.7814994Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:49.7815904Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:49.7816918Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:49.7817731Z tracer.run() 2024-08-20T22:26:49.7818591Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:49.7819553Z while self.step(): 2024-08-20T22:26:49.7820394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:49.7821246Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:49.7822235Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:49.7823067Z return inner_fn(self, inst) 2024-08-20T22:26:49.7824039Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:26:49.7824936Z self.call_function(fn, args, kwargs) 2024-08-20T22:26:49.7825920Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:49.7827049Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:49.7828241Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:49.7829161Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:49.7830207Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:49.7831397Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:49.7832581Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:49.7833650Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:49.7834807Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:49.7835675Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:49.7836653Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:49.7837448Z tracer.run() 2024-08-20T22:26:49.7838182Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:49.7838922Z while self.step(): 2024-08-20T22:26:49.7839698Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:49.7840502Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:49.7841406Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:49.7842170Z return inner_fn(self, inst) 2024-08-20T22:26:49.7842986Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:26:49.7843846Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:26:49.7844912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:26:49.7845725Z return handler(tx, args, kwargs) 2024-08-20T22:26:49.7846705Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:26:49.7847755Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:26:49.7848712Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:26:49.7849680Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:26:49.7850798Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:26:49.7851851Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:26:49.7852921Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:26:49.7853718Z raise UserError( # noqa: B904 2024-08-20T22:26:49.7856729Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:49.7859131Z 2024-08-20T22:26:49.7859405Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:49.7860457Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:49.7861257Z return node.target(*args, **kwargs) 2024-08-20T22:26:49.7861547Z 2024-08-20T22:26:49.7861748Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:49.7862529Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:49.7863504Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7864726Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:49.7865550Z 2024-08-20T22:26:49.7865696Z User Stack (most recent call last): 2024-08-20T22:26:49.7866124Z (snipped, see stack below for prefix) 2024-08-20T22:26:49.7867106Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:49.7868002Z return self.inference(batched_inputs) 2024-08-20T22:26:49.7869173Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:49.7870126Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:49.7871314Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:49.7872222Z images = ImageList.from_tensors( 2024-08-20T22:26:49.7873190Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:49.7874438Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:49.7874853Z 2024-08-20T22:26:49.7875101Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:49.7876281Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:26:49.7877107Z 2024-08-20T22:26:49.7877212Z from user code: 2024-08-20T22:26:49.7878105Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:49.7879001Z return self.inference(batched_inputs) 2024-08-20T22:26:49.7880092Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:49.7881030Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:49.7882133Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:49.7883066Z images = ImageList.from_tensors( 2024-08-20T22:26:49.7884050Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:49.7885135Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:49.7885557Z 2024-08-20T22:26:49.7885950Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:26:49.7886482Z 2024-08-20T22:26:49.7886771Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:26:49.7887274Z fail_to_run 2024-08-20T22:26:49.7887835Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:49.7888488Z TIMING: 2024-08-20T22:26:49.7889009Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:26:49.7889817Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:26:52.9543418Z 2024-08-20T22:27:01.7035954Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:27:01.7036738Z loading model: 0it [00:08, ?it/s] 2024-08-20T22:27:01.7056997Z cpu eval detectron2_maskrcnn_r_101_fpn 2024-08-20T22:27:18.5834100Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_101_fpn. Setting accuracy check to cosine 2024-08-20T22:27:18.6094361Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:26.6421848Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:34.5538748Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:38.9701222Z W0820 22:27:38.969451 17773 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:27:38.9703520Z E0820 22:27:38.969609 17773 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:27:38.9963897Z ERROR:common: 2024-08-20T22:27:38.9965361Z Traceback (most recent call last): 2024-08-20T22:27:38.9966380Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:38.9967192Z return node.target(*args, **kwargs) 2024-08-20T22:27:38.9968455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:27:38.9969422Z r = self.shape_env.evaluate_expr( 2024-08-20T22:27:38.9970414Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:27:38.9971283Z return retlog(event.run(self)) 2024-08-20T22:27:38.9972191Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:27:38.9973033Z return self.f(*args, **kwargs) 2024-08-20T22:27:38.9974077Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:27:38.9975403Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:27:38.9976850Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:27:38.9977798Z raise self._make_data_dependent_error( 2024-08-20T22:27:38.9979416Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:38.9980454Z 2024-08-20T22:27:38.9980755Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:38.9981795Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:38.9982596Z return node.target(*args, **kwargs) 2024-08-20T22:27:38.9982887Z 2024-08-20T22:27:38.9983086Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:38.9983864Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:38.9984841Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:38.9986067Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:38.9986894Z 2024-08-20T22:27:38.9987041Z User Stack (most recent call last): 2024-08-20T22:27:38.9987465Z (snipped, see stack below for prefix) 2024-08-20T22:27:38.9988451Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:38.9989333Z return self.inference(batched_inputs) 2024-08-20T22:27:38.9990328Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:38.9991262Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:38.9992343Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:38.9993274Z images = ImageList.from_tensors( 2024-08-20T22:27:38.9994270Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:38.9995363Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:38.9995773Z 2024-08-20T22:27:38.9996022Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:38.9996439Z 2024-08-20T22:27:38.9996722Z The above exception was the direct cause of the following exception: 2024-08-20T22:27:38.9997180Z 2024-08-20T22:27:38.9997312Z Traceback (most recent call last): 2024-08-20T22:27:38.9998225Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:27:38.9999049Z ret_val = wrap_fake_exception( 2024-08-20T22:27:38.9999969Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:27:39.0000790Z return fn() 2024-08-20T22:27:39.0001619Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:27:39.0002513Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:27:39.0003488Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:27:39.0004368Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:27:39.0005352Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:39.0006138Z return node.target(*args, **kwargs) 2024-08-20T22:27:39.0007174Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:27:39.0008109Z r = self.shape_env.evaluate_expr( 2024-08-20T22:27:39.0009074Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:27:39.0009944Z return retlog(event.run(self)) 2024-08-20T22:27:39.0010849Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:27:39.0011748Z return self.f(*args, **kwargs) 2024-08-20T22:27:39.0012772Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:27:39.0014089Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:27:39.0015532Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:27:39.0016488Z raise self._make_data_dependent_error( 2024-08-20T22:27:39.0017754Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:27:39.0019383Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:39.0020018Z 2024-08-20T22:27:39.0020298Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:39.0021315Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:39.0022113Z return node.target(*args, **kwargs) 2024-08-20T22:27:39.0022411Z 2024-08-20T22:27:39.0022599Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:39.0023381Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:39.0024348Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:39.0025551Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:39.0026380Z 2024-08-20T22:27:39.0026511Z User Stack (most recent call last): 2024-08-20T22:27:39.0026932Z (snipped, see stack below for prefix) 2024-08-20T22:27:39.0027919Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:39.0028803Z return self.inference(batched_inputs) 2024-08-20T22:27:39.0029806Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:39.0030851Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:39.0031912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:39.0032805Z images = ImageList.from_tensors( 2024-08-20T22:27:39.0033774Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:39.0035011Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:39.0035424Z 2024-08-20T22:27:39.0035805Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:39.0036223Z 2024-08-20T22:27:39.0036510Z During handling of the above exception, another exception occurred: 2024-08-20T22:27:39.0036966Z 2024-08-20T22:27:39.0037098Z Traceback (most recent call last): 2024-08-20T22:27:39.0037813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:27:39.0038652Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:27:39.0039426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:27:39.0040346Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:27:39.0041173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:27:39.0041866Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:27:39.0042856Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:27:39.0043888Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:27:39.0044878Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:27:39.0045683Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:27:39.0046683Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:27:39.0047587Z return self._call_impl(*args, **kwargs) 2024-08-20T22:27:39.0048534Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:27:39.0049372Z return forward_call(*args, **kwargs) 2024-08-20T22:27:39.0050241Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:27:39.0051009Z return fn(*args, **kwargs) 2024-08-20T22:27:39.0051955Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:27:39.0052854Z return self._call_impl(*args, **kwargs) 2024-08-20T22:27:39.0053794Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:27:39.0054809Z return forward_call(*args, **kwargs) 2024-08-20T22:27:39.0055745Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:27:39.0056597Z return self._torchdynamo_orig_callable( 2024-08-20T22:27:39.0057528Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:27:39.0058329Z return _compile( 2024-08-20T22:27:39.0059246Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:27:39.0060198Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:27:39.0061283Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:27:39.0062232Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:27:39.0063257Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:27:39.0064094Z return function(*args, **kwargs) 2024-08-20T22:27:39.0065045Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:27:39.0065949Z out_code = transform_code_object(code, transform) 2024-08-20T22:27:39.0067097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:27:39.0068105Z transformations(instructions, code_options) 2024-08-20T22:27:39.0069158Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:27:39.0069943Z return fn(*args, **kwargs) 2024-08-20T22:27:39.0070905Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:27:39.0071655Z tracer.run() 2024-08-20T22:27:39.0087317Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:27:39.0088172Z super().run() 2024-08-20T22:27:39.0089022Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:39.0089802Z while self.step(): 2024-08-20T22:27:39.0090628Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:39.0091476Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:39.0092431Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:39.0093259Z return inner_fn(self, inst) 2024-08-20T22:27:39.0094189Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:27:39.0095252Z self.call_function(fn, args, {}) 2024-08-20T22:27:39.0096187Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:39.0097288Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:39.0098545Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:27:39.0099647Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:39.0100692Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:39.0101630Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:39.0102690Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:39.0103794Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:39.0105074Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:39.0106206Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:39.0107402Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:39.0108334Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:39.0109382Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:39.0110217Z tracer.run() 2024-08-20T22:27:39.0111141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:39.0111917Z while self.step(): 2024-08-20T22:27:39.0112739Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:39.0113574Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:39.0114531Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:39.0115341Z return inner_fn(self, inst) 2024-08-20T22:27:39.0116262Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:27:39.0117101Z self.call_function(fn, args, {}) 2024-08-20T22:27:39.0118109Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:39.0119150Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:39.0120319Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:27:39.0121192Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:39.0122152Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:39.0123018Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:39.0123987Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:39.0124999Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:39.0126163Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:39.0127238Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:39.0128344Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:39.0129261Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:39.0130235Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:39.0130999Z tracer.run() 2024-08-20T22:27:39.0131744Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:39.0132483Z while self.step(): 2024-08-20T22:27:39.0133255Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:39.0134047Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:39.0134955Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:39.0135896Z return inner_fn(self, inst) 2024-08-20T22:27:39.0136845Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:27:39.0137717Z self.call_function(fn, args, kwargs) 2024-08-20T22:27:39.0138768Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:39.0140061Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:39.0141245Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:39.0142173Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:39.0143225Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:39.0144312Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:39.0145596Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:39.0146759Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:39.0147934Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:39.0148867Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:39.0149912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:39.0150758Z tracer.run() 2024-08-20T22:27:39.0151623Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:39.0152366Z while self.step(): 2024-08-20T22:27:39.0153143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:39.0155550Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:39.0156585Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:39.0157433Z return inner_fn(self, inst) 2024-08-20T22:27:39.0158321Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:27:39.0159254Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:27:39.0160340Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:27:39.0161222Z return handler(tx, args, kwargs) 2024-08-20T22:27:39.0162291Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:27:39.0163251Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:27:39.0164234Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:27:39.0165230Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:27:39.0166546Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:27:39.0167699Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:27:39.0168719Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:27:39.0169463Z raise UserError( # noqa: B904 2024-08-20T22:27:39.0171971Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:39.0174083Z 2024-08-20T22:27:39.0174333Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:39.0175297Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:39.0176037Z return node.target(*args, **kwargs) 2024-08-20T22:27:39.0176303Z 2024-08-20T22:27:39.0176492Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:39.0177211Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:39.0178112Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:39.0179517Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:39.0180341Z 2024-08-20T22:27:39.0180489Z User Stack (most recent call last): 2024-08-20T22:27:39.0180919Z (snipped, see stack below for prefix) 2024-08-20T22:27:39.0181904Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:39.0182801Z return self.inference(batched_inputs) 2024-08-20T22:27:39.0183809Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:39.0184743Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:39.0185824Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:39.0186758Z images = ImageList.from_tensors( 2024-08-20T22:27:39.0187747Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:39.0188842Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:39.0189248Z 2024-08-20T22:27:39.0189592Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:39.0190774Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:27:39.0191679Z 2024-08-20T22:27:39.0191778Z from user code: 2024-08-20T22:27:39.0192595Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:39.0193419Z return self.inference(batched_inputs) 2024-08-20T22:27:39.0194335Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:39.0195201Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:39.0196212Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:39.0197072Z images = ImageList.from_tensors( 2024-08-20T22:27:39.0197978Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:39.0198980Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:39.0199426Z 2024-08-20T22:27:39.0199695Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:27:39.0200108Z 2024-08-20T22:27:39.0200390Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:27:39.0200907Z fail_to_run 2024-08-20T22:27:39.0201453Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:39.0202107Z TIMING: 2024-08-20T22:27:39.0202633Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:43 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:27:39.0203424Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:27:42.1840040Z 2024-08-20T22:27:57.3319542Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:27:57.3320069Z loading model: 0it [00:15, ?it/s] 2024-08-20T22:27:57.3328275Z cpu eval detectron2_maskrcnn_r_50_c4 2024-08-20T22:28:05.7625001Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_50_c4. Setting accuracy check to cosine 2024-08-20T22:28:05.7698330Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:29.3460413Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:52.8121876Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:05.2028979Z W0820 22:29:05.202391 18100 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:29:05.2031319Z E0820 22:29:05.202558 18100 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:29:05.2301490Z ERROR:common: 2024-08-20T22:29:05.2301874Z Traceback (most recent call last): 2024-08-20T22:29:05.2302866Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:05.2303937Z return node.target(*args, **kwargs) 2024-08-20T22:29:05.2305640Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:29:05.2306862Z r = self.shape_env.evaluate_expr( 2024-08-20T22:29:05.2308273Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:29:05.2309402Z return retlog(event.run(self)) 2024-08-20T22:29:05.2310730Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:29:05.2311941Z return self.f(*args, **kwargs) 2024-08-20T22:29:05.2313896Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:29:05.2315233Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:29:05.2316660Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:29:05.2317631Z raise self._make_data_dependent_error( 2024-08-20T22:29:05.2318977Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:05.2320007Z 2024-08-20T22:29:05.2320289Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:05.2321321Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:05.2322114Z return node.target(*args, **kwargs) 2024-08-20T22:29:05.2322417Z 2024-08-20T22:29:05.2322605Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:05.2323540Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:05.2324522Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2325724Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:05.2326560Z 2024-08-20T22:29:05.2326695Z User Stack (most recent call last): 2024-08-20T22:29:05.2327120Z (snipped, see stack below for prefix) 2024-08-20T22:29:05.2328126Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:05.2329011Z return self.inference(batched_inputs) 2024-08-20T22:29:05.2330016Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:05.2330950Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:05.2332124Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:05.2333055Z images = ImageList.from_tensors( 2024-08-20T22:29:05.2334049Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:05.2335140Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:05.2335553Z 2024-08-20T22:29:05.2335799Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2336214Z 2024-08-20T22:29:05.2336503Z The above exception was the direct cause of the following exception: 2024-08-20T22:29:05.2336963Z 2024-08-20T22:29:05.2337097Z Traceback (most recent call last): 2024-08-20T22:29:05.2338009Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:29:05.2338943Z ret_val = wrap_fake_exception( 2024-08-20T22:29:05.2339876Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:29:05.2340703Z return fn() 2024-08-20T22:29:05.2341478Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:29:05.2342375Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:29:05.2343346Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:29:05.2344235Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:29:05.2345221Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:05.2346011Z return node.target(*args, **kwargs) 2024-08-20T22:29:05.2347134Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:29:05.2348080Z r = self.shape_env.evaluate_expr( 2024-08-20T22:29:05.2349062Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:29:05.2349938Z return retlog(event.run(self)) 2024-08-20T22:29:05.2350847Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:29:05.2351689Z return self.f(*args, **kwargs) 2024-08-20T22:29:05.2352720Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:29:05.2354041Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:29:05.2355685Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:29:05.2356649Z raise self._make_data_dependent_error( 2024-08-20T22:29:05.2357937Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:29:05.2359586Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:05.2360217Z 2024-08-20T22:29:05.2360505Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:05.2361523Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:05.2362329Z return node.target(*args, **kwargs) 2024-08-20T22:29:05.2362636Z 2024-08-20T22:29:05.2362826Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:05.2363619Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:05.2364587Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2365807Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:05.2366639Z 2024-08-20T22:29:05.2366770Z User Stack (most recent call last): 2024-08-20T22:29:05.2367189Z (snipped, see stack below for prefix) 2024-08-20T22:29:05.2368182Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:05.2369057Z return self.inference(batched_inputs) 2024-08-20T22:29:05.2370056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:05.2370986Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:05.2372083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:05.2373004Z images = ImageList.from_tensors( 2024-08-20T22:29:05.2373992Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:05.2375078Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:05.2375485Z 2024-08-20T22:29:05.2375742Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2376144Z 2024-08-20T22:29:05.2376438Z During handling of the above exception, another exception occurred: 2024-08-20T22:29:05.2376879Z 2024-08-20T22:29:05.2377010Z Traceback (most recent call last): 2024-08-20T22:29:05.2377720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:29:05.2378547Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:29:05.2379346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:29:05.2380378Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:29:05.2381238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:29:05.2381966Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:29:05.2382975Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:29:05.2383843Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:29:05.2384761Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:29:05.2385573Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:29:05.2386585Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:29:05.2387476Z return self._call_impl(*args, **kwargs) 2024-08-20T22:29:05.2388430Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:29:05.2389270Z return forward_call(*args, **kwargs) 2024-08-20T22:29:05.2390143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:29:05.2391041Z return fn(*args, **kwargs) 2024-08-20T22:29:05.2392081Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:29:05.2392964Z return self._call_impl(*args, **kwargs) 2024-08-20T22:29:05.2393884Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:29:05.2394705Z return forward_call(*args, **kwargs) 2024-08-20T22:29:05.2395601Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:29:05.2396423Z return self._torchdynamo_orig_callable( 2024-08-20T22:29:05.2397344Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:29:05.2398123Z return _compile( 2024-08-20T22:29:05.2398918Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:29:05.2399845Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:29:05.2400909Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:29:05.2401828Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:29:05.2402812Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:29:05.2403622Z return function(*args, **kwargs) 2024-08-20T22:29:05.2404622Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:29:05.2405474Z out_code = transform_code_object(code, transform) 2024-08-20T22:29:05.2406711Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:29:05.2407699Z transformations(instructions, code_options) 2024-08-20T22:29:05.2408617Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:29:05.2409391Z return fn(*args, **kwargs) 2024-08-20T22:29:05.2410250Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:29:05.2411026Z tracer.run() 2024-08-20T22:29:05.2411812Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:29:05.2412590Z super().run() 2024-08-20T22:29:05.2413373Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:05.2414135Z while self.step(): 2024-08-20T22:29:05.2415029Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:05.2415884Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:05.2416841Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:05.2417643Z return inner_fn(self, inst) 2024-08-20T22:29:05.2418832Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:29:05.2419726Z self.call_function(fn, args, {}) 2024-08-20T22:29:05.2420700Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:05.2421813Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:05.2423007Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:29:05.2423947Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:05.2425076Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:05.2425990Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:05.2427034Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:05.2428129Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:05.2429401Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:05.2430551Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:05.2431875Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:05.2432743Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:05.2433714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:05.2434498Z tracer.run() 2024-08-20T22:29:05.2435230Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:05.2435968Z while self.step(): 2024-08-20T22:29:05.2436745Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:05.2437550Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:05.2438448Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:05.2439213Z return inner_fn(self, inst) 2024-08-20T22:29:05.2440091Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:29:05.2440900Z self.call_function(fn, args, {}) 2024-08-20T22:29:05.2441778Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:05.2442814Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:05.2443908Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:29:05.2444954Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:05.2445966Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:05.2446847Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:05.2447862Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:05.2448973Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:05.2450221Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:05.2451335Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:05.2452490Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:05.2453401Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:05.2454831Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:05.2455680Z tracer.run() 2024-08-20T22:29:05.2456484Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:05.2457286Z while self.step(): 2024-08-20T22:29:05.2458128Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:05.2459069Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:05.2460162Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:05.2460993Z return inner_fn(self, inst) 2024-08-20T22:29:05.2461965Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:29:05.2462876Z self.call_function(fn, args, kwargs) 2024-08-20T22:29:05.2463847Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:05.2464970Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:05.2466161Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:05.2467091Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:05.2468131Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:05.2469220Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:05.2470602Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:05.2471731Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:05.2472883Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:05.2473777Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:05.2474792Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:05.2475616Z tracer.run() 2024-08-20T22:29:05.2476398Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:05.2477168Z while self.step(): 2024-08-20T22:29:05.2477977Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:05.2478814Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:05.2479763Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:05.2480559Z return inner_fn(self, inst) 2024-08-20T22:29:05.2481412Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:29:05.2482305Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:29:05.2483370Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:29:05.2484303Z return handler(tx, args, kwargs) 2024-08-20T22:29:05.2485336Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:29:05.2486270Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:29:05.2487220Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:29:05.2488191Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:29:05.2489293Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:29:05.2490506Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:29:05.2491596Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:29:05.2492414Z raise UserError( # noqa: B904 2024-08-20T22:29:05.2495112Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:05.2497502Z 2024-08-20T22:29:05.2497785Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:05.2498885Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:05.2499679Z return node.target(*args, **kwargs) 2024-08-20T22:29:05.2499987Z 2024-08-20T22:29:05.2500174Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:05.2500964Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:05.2501925Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2503141Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:05.2503980Z 2024-08-20T22:29:05.2504111Z User Stack (most recent call last): 2024-08-20T22:29:05.2504532Z (snipped, see stack below for prefix) 2024-08-20T22:29:05.2505525Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:05.2506409Z return self.inference(batched_inputs) 2024-08-20T22:29:05.2507412Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:05.2508349Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:05.2509436Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:05.2510351Z images = ImageList.from_tensors( 2024-08-20T22:29:05.2511443Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:05.2512503Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:05.2512901Z 2024-08-20T22:29:05.2513154Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:05.2514289Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:29:05.2515087Z 2024-08-20T22:29:05.2515187Z from user code: 2024-08-20T22:29:05.2516033Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:05.2516899Z return self.inference(batched_inputs) 2024-08-20T22:29:05.2517936Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:05.2518835Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:05.2519896Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:05.2520806Z images = ImageList.from_tensors( 2024-08-20T22:29:05.2521772Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:05.2522817Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:05.2523228Z 2024-08-20T22:29:05.2523510Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:29:05.2523956Z 2024-08-20T22:29:05.2524237Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:29:05.2524777Z fail_to_run 2024-08-20T22:29:05.2525359Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:05.2526037Z TIMING: 2024-08-20T22:29:05.2526586Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:40 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:29:05.2527501Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:29:08.3535241Z 2024-08-20T22:29:15.5268049Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:29:15.5268529Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:29:15.5284687Z cpu eval detectron2_maskrcnn_r_50_fpn 2024-08-20T22:29:30.0880605Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_50_fpn. Setting accuracy check to cosine 2024-08-20T22:29:30.1207948Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:38.9331281Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:47.7360276Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:52.8336394Z W0820 22:29:52.832766 18427 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:29:52.8338907Z E0820 22:29:52.832924 18427 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:29:52.8596509Z ERROR:common: 2024-08-20T22:29:52.8596875Z Traceback (most recent call last): 2024-08-20T22:29:52.8597935Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:52.8600960Z return node.target(*args, **kwargs) 2024-08-20T22:29:52.8602517Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:29:52.8603599Z r = self.shape_env.evaluate_expr( 2024-08-20T22:29:52.8604625Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:29:52.8605502Z return retlog(event.run(self)) 2024-08-20T22:29:52.8606720Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:29:52.8608045Z return self.f(*args, **kwargs) 2024-08-20T22:29:52.8609122Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:29:52.8611120Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:29:52.8613146Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:29:52.8614601Z raise self._make_data_dependent_error( 2024-08-20T22:29:52.8616128Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:52.8617178Z 2024-08-20T22:29:52.8617450Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:52.8618581Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:52.8619384Z return node.target(*args, **kwargs) 2024-08-20T22:29:52.8619672Z 2024-08-20T22:29:52.8619877Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:52.8620659Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:52.8621633Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8622858Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:52.8623775Z 2024-08-20T22:29:52.8623910Z User Stack (most recent call last): 2024-08-20T22:29:52.8624344Z (snipped, see stack below for prefix) 2024-08-20T22:29:52.8625350Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:52.8626370Z return self.inference(batched_inputs) 2024-08-20T22:29:52.8627365Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:52.8628300Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:52.8629394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:52.8630330Z images = ImageList.from_tensors( 2024-08-20T22:29:52.8631312Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:52.8632403Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:52.8632824Z 2024-08-20T22:29:52.8633077Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8633477Z 2024-08-20T22:29:52.8633778Z The above exception was the direct cause of the following exception: 2024-08-20T22:29:52.8634230Z 2024-08-20T22:29:52.8634373Z Traceback (most recent call last): 2024-08-20T22:29:52.8635264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:29:52.8636087Z ret_val = wrap_fake_exception( 2024-08-20T22:29:52.8637017Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:29:52.8637839Z return fn() 2024-08-20T22:29:52.8638596Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:29:52.8639478Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:29:52.8640469Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:29:52.8641352Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:29:52.8642320Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:52.8643115Z return node.target(*args, **kwargs) 2024-08-20T22:29:52.8644167Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:29:52.8645109Z r = self.shape_env.evaluate_expr( 2024-08-20T22:29:52.8646056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:29:52.8646915Z return retlog(event.run(self)) 2024-08-20T22:29:52.8647827Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:29:52.8648658Z return self.f(*args, **kwargs) 2024-08-20T22:29:52.8649745Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:29:52.8651057Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:29:52.8652501Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:29:52.8653475Z raise self._make_data_dependent_error( 2024-08-20T22:29:52.8654925Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:29:52.8656463Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:52.8657118Z 2024-08-20T22:29:52.8657387Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:52.8658419Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:52.8659281Z return node.target(*args, **kwargs) 2024-08-20T22:29:52.8659685Z 2024-08-20T22:29:52.8659888Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:52.8660668Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:52.8661638Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8662853Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:52.8663679Z 2024-08-20T22:29:52.8663825Z User Stack (most recent call last): 2024-08-20T22:29:52.8664231Z (snipped, see stack below for prefix) 2024-08-20T22:29:52.8665227Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:52.8666123Z return self.inference(batched_inputs) 2024-08-20T22:29:52.8667121Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:52.8668036Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:52.8669131Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:52.8670072Z images = ImageList.from_tensors( 2024-08-20T22:29:52.8671161Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:52.8672219Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:52.8672615Z 2024-08-20T22:29:52.8672854Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8673260Z 2024-08-20T22:29:52.8673536Z During handling of the above exception, another exception occurred: 2024-08-20T22:29:52.8673981Z 2024-08-20T22:29:52.8674111Z Traceback (most recent call last): 2024-08-20T22:29:52.8674807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:29:52.8675538Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:29:52.8676302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:29:52.8677231Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:29:52.8678061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:29:52.8678750Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:29:52.8679726Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:29:52.8680585Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:29:52.8681481Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:29:52.8682368Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:29:52.8683347Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:29:52.8684238Z return self._call_impl(*args, **kwargs) 2024-08-20T22:29:52.8685170Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:29:52.8685993Z return forward_call(*args, **kwargs) 2024-08-20T22:29:52.8686838Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:29:52.8687848Z return fn(*args, **kwargs) 2024-08-20T22:29:52.8688776Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:29:52.8689657Z return self._call_impl(*args, **kwargs) 2024-08-20T22:29:52.8690572Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:29:52.8691393Z return forward_call(*args, **kwargs) 2024-08-20T22:29:52.8692297Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:29:52.8693190Z return self._torchdynamo_orig_callable( 2024-08-20T22:29:52.8694097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:29:52.8694873Z return _compile( 2024-08-20T22:29:52.8695687Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:29:52.8696616Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:29:52.8697673Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:29:52.8698669Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:29:52.8699878Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:29:52.8700717Z return function(*args, **kwargs) 2024-08-20T22:29:52.8701687Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:29:52.8702596Z out_code = transform_code_object(code, transform) 2024-08-20T22:29:52.8703751Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:29:52.8704763Z transformations(instructions, code_options) 2024-08-20T22:29:52.8705709Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:29:52.8706483Z return fn(*args, **kwargs) 2024-08-20T22:29:52.8707376Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:29:52.8708184Z tracer.run() 2024-08-20T22:29:52.8709004Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:29:52.8709796Z super().run() 2024-08-20T22:29:52.8710610Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:52.8711514Z while self.step(): 2024-08-20T22:29:52.8712334Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:52.8713161Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:52.8714117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:52.8714937Z return inner_fn(self, inst) 2024-08-20T22:29:52.8715928Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:29:52.8716724Z self.call_function(fn, args, {}) 2024-08-20T22:29:52.8717681Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:52.8718737Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:52.8719838Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:29:52.8720703Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:52.8721661Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:52.8722530Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:52.8723502Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:52.8724507Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:52.8725678Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:52.8726817Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:52.8727915Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:52.8728787Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:52.8729746Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:52.8730532Z tracer.run() 2024-08-20T22:29:52.8731281Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:52.8732013Z while self.step(): 2024-08-20T22:29:52.8732789Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:52.8733581Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:52.8734491Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:52.8735259Z return inner_fn(self, inst) 2024-08-20T22:29:52.8736141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:29:52.8736936Z self.call_function(fn, args, {}) 2024-08-20T22:29:52.8737831Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:52.8739144Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:52.8740333Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:29:52.8741251Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:52.8742317Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:52.8743249Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:52.8744296Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:52.8745363Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:52.8746642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:52.8747785Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:52.8748963Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:52.8749904Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:52.8751055Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:52.8752001Z tracer.run() 2024-08-20T22:29:52.8752877Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:52.8753613Z while self.step(): 2024-08-20T22:29:52.8754527Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:52.8755562Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:52.8756551Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:52.8757382Z return inner_fn(self, inst) 2024-08-20T22:29:52.8758342Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:29:52.8759247Z self.call_function(fn, args, kwargs) 2024-08-20T22:29:52.8760227Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:29:52.8761351Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:29:52.8762663Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:29:52.8763593Z return super().call_function(tx, args, kwargs) 2024-08-20T22:29:52.8764648Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:29:52.8765737Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:29:52.8767133Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:29:52.8768189Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:29:52.8769297Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:29:52.8770159Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:29:52.8771131Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:29:52.8771898Z tracer.run() 2024-08-20T22:29:52.8772642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:29:52.8773383Z while self.step(): 2024-08-20T22:29:52.8774152Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:29:52.8774950Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:29:52.8776049Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:29:52.8776844Z return inner_fn(self, inst) 2024-08-20T22:29:52.8777709Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:29:52.8778702Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:29:52.8779965Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:29:52.8780843Z return handler(tx, args, kwargs) 2024-08-20T22:29:52.8781916Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:29:52.8782872Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:29:52.8783836Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:29:52.8784835Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:29:52.8786072Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:29:52.8787158Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:29:52.8788251Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:29:52.8789075Z raise UserError( # noqa: B904 2024-08-20T22:29:52.8791864Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:29:52.8793976Z 2024-08-20T22:29:52.8794229Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:29:52.8795196Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:29:52.8795941Z return node.target(*args, **kwargs) 2024-08-20T22:29:52.8796208Z 2024-08-20T22:29:52.8797748Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:29:52.8798704Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:29:52.8799680Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8800902Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:29:52.8801729Z 2024-08-20T22:29:52.8801880Z User Stack (most recent call last): 2024-08-20T22:29:52.8802292Z (snipped, see stack below for prefix) 2024-08-20T22:29:52.8803313Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:52.8804210Z return self.inference(batched_inputs) 2024-08-20T22:29:52.8805231Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:52.8806161Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:52.8807264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:52.8808195Z images = ImageList.from_tensors( 2024-08-20T22:29:52.8809202Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:52.8810477Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:52.8810869Z 2024-08-20T22:29:52.8811099Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:29:52.8812195Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:29:52.8812945Z 2024-08-20T22:29:52.8813055Z from user code: 2024-08-20T22:29:52.8813868Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:29:52.8814684Z return self.inference(batched_inputs) 2024-08-20T22:29:52.8815615Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:29:52.8816469Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:29:52.8817474Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:29:52.8818324Z images = ImageList.from_tensors( 2024-08-20T22:29:52.8819548Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:29:52.8820636Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:29:52.8821043Z 2024-08-20T22:29:52.8821426Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:29:52.8821875Z 2024-08-20T22:29:52.8822180Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:29:52.8822727Z fail_to_run 2024-08-20T22:29:52.8823329Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:52.8824026Z TIMING: 2024-08-20T22:29:52.8824594Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:43 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:29:52.8825463Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:29:55.9657289Z 2024-08-20T22:30:04.3163160Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:04.3164148Z loading model: 0it [00:08, ?it/s] 2024-08-20T22:30:04.3164901Z cpu eval dlrm 2024-08-20T22:30:04.9929499Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:05.2195714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:05.4448328Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:30.9520521Z Error: weights_offset must be aligned to 16K boundary 2024-08-20T22:30:30.9524586Z ERROR:common: 2024-08-20T22:30:30.9524959Z Traceback (most recent call last): 2024-08-20T22:30:30.9525730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:30:30.9529825Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:30:30.9530648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:30:30.9531609Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:30:30.9532522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1380, in load 2024-08-20T22:30:30.9533469Z cls.cache[key] = torch._export.aot_load(so_path, device) 2024-08-20T22:30:30.9534670Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_export/__init__.py", line 300, in aot_load 2024-08-20T22:30:30.9535861Z runner = torch._C._aoti.AOTIModelContainerRunnerCpu(so_path, 1) # type: ignore[call-arg] 2024-08-20T22:30:30.9537638Z RuntimeError: create_func_( &container_handle_, num_models, device_str.c_str(), cubin_dir.empty() ? nullptr : cubin_dir.c_str()) API call failed at /var/lib/jenkins/workspace/torch/csrc/inductor/aoti_runner/model_container_runner.cpp, line 70 2024-08-20T22:30:30.9539439Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:30:30.9539993Z fail_to_run 2024-08-20T22:30:30.9540584Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:30.9541533Z TIMING: backend_compile:0.0001 entire_frame_compile:0.26498 total_wall_time:0.26498 2024-08-20T22:30:30.9542724Z STATS: call_* op count: 36 | FakeTensor.__torch_dispatch__:187 | FakeTensorMode.__torch_dispatch__:2410 | ProxyTorchDispatchMode.__torch_dispatch__:546 2024-08-20T22:30:30.9543853Z Dynamo produced 1 graphs covering 36 ops with 0 graph breaks (0 unique) 2024-08-20T22:30:34.4513133Z 2024-08-20T22:30:35.6852861Z loading model: 0it [00:00, ?it/s]Downloading https://doctr-static.mindee.com/models?id=v0.7.0/db_resnet50-79bd7d70.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/db_resnet50-79bd7d70.pt 2024-08-20T22:30:36.0231959Z 2024-08-20T22:30:36.0232098Z 2024-08-20T22:30:36.1230174Z 0% 0/102021912 [00:00 2024-08-20T22:31:13.7100344Z [ 2024-08-20T22:31:13.7101133Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/detection/core.py", line 92, in 2024-08-20T22:31:13.7102144Z cv2.morphologyEx(bmap[..., idx], cv2.MORPH_OPEN, self._opening_kernel) 2024-08-20T22:31:13.7102610Z 2024-08-20T22:31:13.7102900Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:31:13.7103347Z 2024-08-20T22:31:13.7103654Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:31:13.7104212Z fail_to_run 2024-08-20T22:31:13.7104805Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:13.7105509Z TIMING: 2024-08-20T22:31:13.7106172Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:605 | FakeTensorMode.__torch_dispatch__:3780 2024-08-20T22:31:13.7107070Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:31:16.8735572Z 2024-08-20T22:31:20.3006370Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:20.3007152Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:31:20.3013679Z cpu eval doctr_reco_predictor 2024-08-20T22:31:20.5275280Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:20.5726889Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:20.6123653Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:22.2987670Z W0820 22:31:22.298283 19481 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Ne(u0, 123), hint=None, expect_rational=True, size_oblivious=False, forcing_spec=False 2024-08-20T22:31:22.2989819Z E0820 22:31:22.298433 19481 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne}) 2024-08-20T22:31:22.3063292Z ERROR:common: 2024-08-20T22:31:22.3063639Z Traceback (most recent call last): 2024-08-20T22:31:22.3064738Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:31:22.3065633Z return guard_scalar(self.sym_num) 2024-08-20T22:31:22.3066689Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 845, in guard_scalar 2024-08-20T22:31:22.3067600Z return guard_bool(a) 2024-08-20T22:31:22.3068571Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1018, in guard_bool 2024-08-20T22:31:22.3069612Z return a.node.guard_bool("", 0) # NB: uses Python backtrace 2024-08-20T22:31:22.3070724Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 451, in guard_bool 2024-08-20T22:31:22.3071778Z r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) 2024-08-20T22:31:22.3072964Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:31:22.3073829Z return retlog(event.run(self)) 2024-08-20T22:31:22.3074832Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:31:22.3075670Z return self.f(*args, **kwargs) 2024-08-20T22:31:22.3076701Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:31:22.3078012Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:31:22.3079441Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5168, in _evaluate_expr 2024-08-20T22:31:22.3080405Z concrete_val = compute_concrete_val() 2024-08-20T22:31:22.3081527Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5149, in compute_concrete_val 2024-08-20T22:31:22.3082520Z return self.size_hint(orig_expr) 2024-08-20T22:31:22.3083549Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4671, in size_hint 2024-08-20T22:31:22.3084548Z raise self._make_data_dependent_error(result_expr, expr) 2024-08-20T22:31:22.3085990Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T22:31:22.3087036Z 2024-08-20T22:31:22.3087303Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:31:22.3088642Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:31:22.3089530Z return guard_scalar(self.sym_num) 2024-08-20T22:31:22.3089822Z 2024-08-20T22:31:22.3090018Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:31:22.3090812Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:31:22.3091786Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:31:22.3093007Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:31:22.3093831Z 2024-08-20T22:31:22.3093963Z User Stack (most recent call last): 2024-08-20T22:31:22.3094386Z (snipped, see stack below for prefix) 2024-08-20T22:31:22.3095421Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:31:22.3096354Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:31:22.3097394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:31:22.3098703Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:31:22.3099999Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:31:22.3100908Z words = [ 2024-08-20T22:31:22.3101810Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3102908Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3104122Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3105224Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3105688Z 2024-08-20T22:31:22.3105952Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:31:22.3106358Z 2024-08-20T22:31:22.3106641Z During handling of the above exception, another exception occurred: 2024-08-20T22:31:22.3107100Z 2024-08-20T22:31:22.3107232Z Traceback (most recent call last): 2024-08-20T22:31:22.3107950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:31:22.3108702Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:31:22.3109480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:31:22.3110423Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:31:22.3111277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:31:22.3111998Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:31:22.3112991Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:31:22.3113873Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:31:22.3114792Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:31:22.3115608Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:31:22.3116619Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:22.3117504Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:22.3118457Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:22.3119299Z return forward_call(*args, **kwargs) 2024-08-20T22:31:22.3120170Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:31:22.3120924Z return fn(*args, **kwargs) 2024-08-20T22:31:22.3121956Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:22.3122859Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:22.3123815Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:22.3124646Z return forward_call(*args, **kwargs) 2024-08-20T22:31:22.3125578Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:31:22.3126434Z return self._torchdynamo_orig_callable( 2024-08-20T22:31:22.3127380Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:31:22.3128165Z return _compile( 2024-08-20T22:31:22.3129001Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:31:22.3129954Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:31:22.3131050Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:31:22.3132032Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:31:22.3133059Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:31:22.3133882Z return function(*args, **kwargs) 2024-08-20T22:31:22.3134840Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:31:22.3135761Z out_code = transform_code_object(code, transform) 2024-08-20T22:31:22.3136891Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:31:22.3137895Z transformations(instructions, code_options) 2024-08-20T22:31:22.3138918Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:31:22.3139716Z return fn(*args, **kwargs) 2024-08-20T22:31:22.3140594Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:31:22.3141403Z tracer.run() 2024-08-20T22:31:22.3142216Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:31:22.3143012Z super().run() 2024-08-20T22:31:22.3143804Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:31:22.3144596Z while self.step(): 2024-08-20T22:31:22.3145427Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:31:22.3146291Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:31:22.3147260Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:31:22.3148088Z return inner_fn(self, inst) 2024-08-20T22:31:22.3149037Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:31:22.3149913Z self.call_function(fn, args, {}) 2024-08-20T22:31:22.3150956Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:31:22.3151992Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:31:22.3153109Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/user_defined.py", line 876, in call_function 2024-08-20T22:31:22.3154015Z return self.call_method(tx, "__call__", args, kwargs) 2024-08-20T22:31:22.3155388Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/user_defined.py", line 736, in call_method 2024-08-20T22:31:22.3156440Z return UserMethodVariable(method, self, source=source).call_function( 2024-08-20T22:31:22.3157734Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:31:22.3158677Z return super().call_function(tx, args, kwargs) 2024-08-20T22:31:22.3159727Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:31:22.3160639Z return super().call_function(tx, args, kwargs) 2024-08-20T22:31:22.3161685Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:31:22.3162782Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:31:22.3164064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:31:22.3165219Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:31:22.3166397Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:31:22.3167495Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:31:22.3168469Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:31:22.3169264Z tracer.run() 2024-08-20T22:31:22.3170002Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:31:22.3170740Z while self.step(): 2024-08-20T22:31:22.3171515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:31:22.3172314Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:31:22.3173212Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:31:22.3173995Z return inner_fn(self, inst) 2024-08-20T22:31:22.3174890Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:31:22.3175725Z self.call_function(fn, args, kwargs) 2024-08-20T22:31:22.3176621Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:31:22.3177657Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:31:22.3179003Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:31:22.3179948Z return super().call_function(tx, args, kwargs) 2024-08-20T22:31:22.3181001Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:31:22.3182085Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:31:22.3183363Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:31:22.3184522Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:31:22.3185708Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:31:22.3186628Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:31:22.3187674Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:31:22.3188530Z tracer.run() 2024-08-20T22:31:22.3189335Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:31:22.3190132Z while self.step(): 2024-08-20T22:31:22.3191065Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:31:22.3191997Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:31:22.3192963Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:31:22.3193778Z return inner_fn(self, inst) 2024-08-20T22:31:22.3194691Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:31:22.3195542Z self.call_function(fn, args, {}) 2024-08-20T22:31:22.3196480Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:31:22.3197566Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:31:22.3198707Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:31:22.3199874Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:31:22.3201052Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:31:22.3202842Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:31:22.3203944Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:31:22.3204807Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:31:22.3205776Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:31:22.3206557Z tracer.run() 2024-08-20T22:31:22.3207302Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:31:22.3208024Z while self.step(): 2024-08-20T22:31:22.3208798Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:31:22.3209609Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:31:22.3210515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:31:22.3211275Z return inner_fn(self, inst) 2024-08-20T22:31:22.3212154Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:31:22.3212965Z self.call_function(fn, args, {}) 2024-08-20T22:31:22.3213852Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:31:22.3214874Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:31:22.3215970Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:31:22.3216987Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:31:22.3218163Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:31:22.3219520Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:31:22.3220697Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:31:22.3221625Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:31:22.3222671Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:31:22.3223515Z tracer.run() 2024-08-20T22:31:22.3224310Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:31:22.3225106Z while self.step(): 2024-08-20T22:31:22.3226010Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:31:22.3226874Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:31:22.3227832Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 532, in inner 2024-08-20T22:31:22.3228711Z eval_result = value.evaluate_expr(self.output) 2024-08-20T22:31:22.3229751Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1105, in evaluate_expr 2024-08-20T22:31:22.3230629Z raise UserError( # noqa: B904 2024-08-20T22:31:22.3232057Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T22:31:22.3233094Z 2024-08-20T22:31:22.3233355Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:31:22.3234465Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:31:22.3235324Z return guard_scalar(self.sym_num) 2024-08-20T22:31:22.3235594Z 2024-08-20T22:31:22.3235851Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:31:22.3236610Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:31:22.3237553Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:31:22.3238736Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:31:22.3239534Z 2024-08-20T22:31:22.3239678Z User Stack (most recent call last): 2024-08-20T22:31:22.3240090Z (snipped, see stack below for prefix) 2024-08-20T22:31:22.3241080Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:31:22.3241998Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:31:22.3243130Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:31:22.3244166Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:31:22.3245346Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:31:22.3246188Z words = [ 2024-08-20T22:31:22.3247025Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3248051Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3249181Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3250184Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3250628Z 2024-08-20T22:31:22.3250857Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:31:22.3251951Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:31:22.3252704Z 2024-08-20T22:31:22.3252812Z from user code: 2024-08-20T22:31:22.3253640Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:31:22.3254818Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:31:22.3255936Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:31:22.3257059Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:31:22.3258417Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:31:22.3259312Z words = [ 2024-08-20T22:31:22.3260332Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3261436Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3262659Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:31:22.3263745Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:31:22.3264227Z 2024-08-20T22:31:22.3264516Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:31:22.3264973Z 2024-08-20T22:31:22.3265262Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:31:22.3265815Z fail_to_run 2024-08-20T22:31:22.3266404Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:22.3267097Z TIMING: 2024-08-20T22:31:22.3267676Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:260 | FakeTensorMode.__torch_dispatch__:5453 2024-08-20T22:31:22.3268570Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:31:25.3730960Z 2024-08-20T22:31:26.1273746Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:26.1274391Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:26.1274964Z cpu eval drq 2024-08-20T22:31:26.1360077Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:26.1422521Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:26.1478670Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:26.5127763Z ERROR:common: 2024-08-20T22:31:26.5128736Z Traceback (most recent call last): 2024-08-20T22:31:26.5129559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:31:26.5130338Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:31:26.5131144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:31:26.5132114Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:31:26.5132988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:31:26.5133716Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:31:26.5134902Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:31:26.5135792Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:31:26.5136733Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1574, in inner 2024-08-20T22:31:26.5137536Z graph = rewrite_signature( 2024-08-20T22:31:26.5138599Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1077, in rewrite_signature 2024-08-20T22:31:26.5139661Z check_user_input_output(flat_results_traced, UserErrorType.INVALID_OUTPUT) 2024-08-20T22:31:26.5140890Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1063, in check_user_input_output 2024-08-20T22:31:26.5141770Z raise UserError( 2024-08-20T22:31:26.5143014Z torch._dynamo.exc.UserError: It looks like one of the outputs with type `` is not supported or pytree-flattenable. 2024-08-20T22:31:26.5146911Z Exported graphs outputs can only contain the following supported types: [, , , , , , , , , , , , , , , , , , ]. 2024-08-20T22:31:26.5150278Z If you are using a custom class object, please register a pytree_flatten/unflatten function using `torch.utils._pytree.register_pytree_node` or `torch.export.register_dataclass`. 2024-08-20T22:31:26.5151610Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:31:26.5152165Z fail_to_run 2024-08-20T22:31:26.5152763Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:26.5153790Z TIMING: backend_compile:8e-05 entire_frame_compile:0.21196 total_wall_time:0.21196 2024-08-20T22:31:26.5154883Z STATS: call_* op count: 29 | FakeTensor.__torch_dispatch__:66 | FakeTensorMode.__torch_dispatch__:219 2024-08-20T22:31:26.5155778Z Dynamo produced 1 graphs covering 29 ops with 0 graph breaks (0 unique) 2024-08-20T22:31:29.5115456Z 2024-08-20T22:31:29.8963412Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:29.8964024Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:29.8967623Z cpu eval functorch_dp_cifar10 2024-08-20T22:31:29.9376684Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:29.9606888Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:29.9828594Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:59.5784722Z pass 2024-08-20T22:31:59.5787529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:59.5788830Z TIMING: backend_compile:0.0001 entire_frame_compile:0.39596 total_wall_time:0.39596 2024-08-20T22:31:59.5790461Z STATS: call_* op count: 69 | FakeTensor.__torch_dispatch__:1297 | FakeTensorMode.__torch_dispatch__:5957 | ProxyTorchDispatchMode.__torch_dispatch__:1200 2024-08-20T22:31:59.5792025Z Dynamo produced 1 graphs covering 69 ops with 0 graph breaks (0 unique) 2024-08-20T22:32:03.0844609Z 2024-08-20T22:32:03.2326388Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:03.2326985Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:03.2327593Z cpu eval functorch_maml_omniglot 2024-08-20T22:32:03.2373280Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:03.2418953Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:03.2453675Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:24.2635050Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_0(const float*, const float*, const float*, float*, float*, float*, float*)’: 2024-08-20T22:32:24.2638264Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp:480:36: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:32:24.2639825Z 480 | Welford tmp_acc0 = Welford(); 2024-08-20T22:32:24.2640351Z | ^~~~~~~~ 2024-08-20T22:32:24.2642401Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_1(const float*, const float*, const float*, float*, float*, float*, float*)’: 2024-08-20T22:32:24.2645456Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp:562:32: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:32:24.2646971Z 562 | Welford tmp_acc0 = Welford(); 2024-08-20T22:32:24.2647711Z | ^~~~~~~~ 2024-08-20T22:32:24.2649873Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_2(const float*, const float*, const float*, float*, float*, float*, float*, float*)’: 2024-08-20T22:32:24.2652941Z /tmp/tmpl_5d7v7j/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/c3lmwij3eysd32y3qyj5lrw7rqub2fa3rtzyghwdnwyhtw4s6tjs.cpp:645:32: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:32:24.2654639Z 645 | Welford tmp_acc0 = Welford(); 2024-08-20T22:32:24.2655187Z | ^~~~~~~~ 2024-08-20T22:32:24.5221676Z pass 2024-08-20T22:32:24.5224703Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:24.5226324Z TIMING: backend_compile:9e-05 entire_frame_compile:0.1609 total_wall_time:0.1609 2024-08-20T22:32:24.5227923Z STATS: call_* op count: 14 | FakeTensor.__torch_dispatch__:208 | FakeTensorMode.__torch_dispatch__:1958 | ProxyTorchDispatchMode.__torch_dispatch__:331 2024-08-20T22:32:24.5229745Z Dynamo produced 1 graphs covering 14 ops with 0 graph breaks (0 unique) 2024-08-20T22:32:27.8389905Z 2024-08-20T22:32:29.2306300Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:29.2306765Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:32:29.2308908Z cpu eval hf_Albert 2024-08-20T22:32:30.5217237Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:31.1490390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:31.8043410Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:23.7602313Z pass 2024-08-20T22:33:23.7605596Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:23.7607618Z TIMING: backend_compile:0.00017 entire_frame_compile:1.64552 total_wall_time:1.64552 2024-08-20T22:33:23.7609305Z STATS: call_* op count: 436 | FakeTensorMode.__torch_dispatch__:19509 | FakeTensor.__torch_dispatch__:3072 | ProxyTorchDispatchMode.__torch_dispatch__:5399 2024-08-20T22:33:23.7610877Z Dynamo produced 1 graphs covering 436 ops with 0 graph breaks (0 unique) 2024-08-20T22:33:27.3821350Z 2024-08-20T22:33:31.3160239Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:31.3160712Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:33:31.3178440Z cpu eval hf_Bart 2024-08-20T22:33:32.9606346Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:33.6052958Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:34.2609446Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:50.0259594Z pass 2024-08-20T22:34:50.0260359Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:50.0261363Z TIMING: backend_compile:0.00015 entire_frame_compile:1.95523 total_wall_time:1.95523 2024-08-20T22:34:50.0262595Z STATS: call_* op count: 451 | FakeTensorMode.__torch_dispatch__:24106 | FakeTensor.__torch_dispatch__:3611 | ProxyTorchDispatchMode.__torch_dispatch__:6144 2024-08-20T22:34:50.0263778Z Dynamo produced 1 graphs covering 451 ops with 0 graph breaks (0 unique) 2024-08-20T22:34:53.7456903Z 2024-08-20T22:34:56.4072468Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:56.4073046Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:34:56.4090872Z cpu eval hf_Bert 2024-08-20T22:34:58.1612438Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:58.7738784Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:59.3756003Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:00.1747278Z pass 2024-08-20T22:36:00.1750963Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:00.1752723Z TIMING: backend_compile:0.00016 entire_frame_compile:3.1805 total_wall_time:3.1805 2024-08-20T22:36:00.1753975Z STATS: call_* op count: 367 | FakeTensorMode.__torch_dispatch__:20747 | FakeTensor.__torch_dispatch__:2855 | ProxyTorchDispatchMode.__torch_dispatch__:5898 2024-08-20T22:36:00.1755324Z Dynamo produced 1 graphs covering 367 ops with 0 graph breaks (0 unique) 2024-08-20T22:36:03.9072614Z 2024-08-20T22:36:09.3766832Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:09.3767315Z loading model: 0it [00:05, ?it/s] 2024-08-20T22:36:09.3796078Z cpu eval hf_Bert_large 2024-08-20T22:36:14.3847666Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:16.0208764Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:17.6402769Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:19.0369992Z pass 2024-08-20T22:38:19.0370748Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:19.0371712Z TIMING: backend_compile:0.00023 entire_frame_compile:6.119 total_wall_time:6.119 2024-08-20T22:38:19.0372929Z STATS: call_* op count: 715 | FakeTensorMode.__torch_dispatch__:40223 | FakeTensor.__torch_dispatch__:5567 | ProxyTorchDispatchMode.__torch_dispatch__:11490 2024-08-20T22:38:19.0374084Z Dynamo produced 1 graphs covering 715 ops with 0 graph breaks (0 unique) 2024-08-20T22:38:23.0575756Z 2024-08-20T22:38:24.9838268Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:38:24.9838782Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:38:24.9842929Z cpu eval hf_DistilBert 2024-08-20T22:38:26.0713489Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:26.6682681Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:27.2187350Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:39:11.8945032Z pass 2024-08-20T22:39:11.8949605Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:39:11.8950759Z TIMING: backend_compile:0.00011 entire_frame_compile:0.90077 total_wall_time:0.90077 2024-08-20T22:39:11.8951982Z STATS: call_* op count: 203 | FakeTensor.__torch_dispatch__:1758 | FakeTensorMode.__torch_dispatch__:11740 | ProxyTorchDispatchMode.__torch_dispatch__:3051 2024-08-20T22:39:11.8953168Z Dynamo produced 1 graphs covering 203 ops with 0 graph breaks (0 unique) 2024-08-20T22:39:15.3817037Z 2024-08-20T22:39:18.3382138Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:18.3382678Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:39:18.3400453Z cpu eval hf_GPT2 2024-08-20T22:39:19.0746171Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:39:19.2901950Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:39:19.5021466Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:40:23.5055795Z pass 2024-08-20T22:40:23.5056565Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:40:23.5057546Z TIMING: backend_compile:0.00021 entire_frame_compile:2.58542 total_wall_time:2.58542 2024-08-20T22:40:23.5059264Z STATS: call_* op count: 622 | FakeTensorMode.__torch_dispatch__:20174 | FakeTensor.__torch_dispatch__:2288 | ProxyTorchDispatchMode.__torch_dispatch__:5631 2024-08-20T22:40:23.5060468Z Dynamo produced 1 graphs covering 622 ops with 0 graph breaks (0 unique) 2024-08-20T22:40:27.1998783Z 2024-08-20T22:40:50.0847944Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:40:50.0848432Z loading model: 0it [00:22, ?it/s] 2024-08-20T22:40:50.0878109Z cpu eval hf_GPT2_large 2024-08-20T22:40:50.0878967Z pass_due_to_skip 2024-08-20T22:40:50.0881471Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:40:50.0883486Z TIMING: 2024-08-20T22:40:50.0884008Z STATS: call_* op count: 0 2024-08-20T22:40:50.0884912Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:40:53.6979876Z 2024-08-20T22:41:08.0619645Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:41:08.0620127Z loading model: 0it [00:14, ?it/s] 2024-08-20T22:41:08.0644156Z cpu eval hf_T5_base 2024-08-20T22:41:37.9495509Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:41:49.1164627Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:42:03.2432748Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:44:26.0108026Z pass 2024-08-20T22:44:26.0110347Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:44:26.0111668Z TIMING: backend_compile:0.00043 entire_frame_compile:6.81795 total_wall_time:6.81795 2024-08-20T22:44:26.0113325Z STATS: call_* op count: 1491 | FakeTensorMode.__torch_dispatch__:49923 | FakeTensor.__torch_dispatch__:7443 | ProxyTorchDispatchMode.__torch_dispatch__:14824 2024-08-20T22:44:26.0114917Z Dynamo produced 1 graphs covering 1491 ops with 0 graph breaks (0 unique) 2024-08-20T22:44:30.7490671Z 2024-08-20T22:44:43.4614712Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:43.4615236Z loading model: 0it [00:12, ?it/s] 2024-08-20T22:44:43.4656171Z cpu eval hf_T5_large 2024-08-20T22:44:43.4657218Z pass_due_to_skip 2024-08-20T22:44:43.4659681Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:44:43.4660395Z TIMING: 2024-08-20T22:44:43.4660662Z STATS: call_* op count: 0 2024-08-20T22:44:43.4661189Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:44:47.0276542Z 2024-08-20T22:44:53.7721378Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:53.7721859Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:44:53.7742471Z cpu eval hf_distil_whisper 2024-08-20T22:45:02.8634464Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:45:06.9843922Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:45:11.1888411Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:46:19.3804876Z /tmp/tmpl_5d7v7j/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/cimeew2inxjq3chopj3ax27kz64rclpu5alpksu5wvkpl4iomhfm.cpp: In function ‘void cpp_fused_add_native_layer_norm_1(const float*, const float*, const float*, const float*, float*, float*, float*, float*)’: 2024-08-20T22:46:19.3809761Z /tmp/tmpl_5d7v7j/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/cimeew2inxjq3chopj3ax27kz64rclpu5alpksu5wvkpl4iomhfm.cpp:512:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:46:19.3811324Z 512 | Welford tmp_acc0 = Welford(); 2024-08-20T22:46:19.3811881Z | ^~~~~~~~ 2024-08-20T22:46:19.3813829Z /tmp/tmpl_5d7v7j/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/cimeew2inxjq3chopj3ax27kz64rclpu5alpksu5wvkpl4iomhfm.cpp:549:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:46:19.3815444Z 549 | Welford tmp_acc0 = Welford(); 2024-08-20T22:46:19.3815979Z | ^~~~~~~~ 2024-08-20T22:47:12.8545600Z pass 2024-08-20T22:47:12.8548692Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:12.8550828Z TIMING: backend_compile:0.00019 entire_frame_compile:2.7047 total_wall_time:2.7047 2024-08-20T22:47:12.8552524Z STATS: call_* op count: 635 | FakeTensor.__torch_dispatch__:5714 | FakeTensorMode.__torch_dispatch__:37670 | ProxyTorchDispatchMode.__torch_dispatch__:9016 2024-08-20T22:47:12.8553961Z Dynamo produced 1 graphs covering 635 ops with 0 graph breaks (0 unique) 2024-08-20T22:47:17.1044944Z 2024-08-20T22:47:17.2204412Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:17.2204915Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:17.2205381Z cpu eval lennard_jones 2024-08-20T22:47:17.2232488Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:17.2254400Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:17.2262551Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:37.5932467Z pass 2024-08-20T22:47:37.5934783Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:37.5936407Z TIMING: backend_compile:8e-05 entire_frame_compile:0.1069 total_wall_time:0.1069 2024-08-20T22:47:37.5938003Z STATS: call_* op count: 9 | FakeTensor.__torch_dispatch__:81 | FakeTensorMode.__torch_dispatch__:973 | ProxyTorchDispatchMode.__torch_dispatch__:218 2024-08-20T22:47:37.5939616Z Dynamo produced 1 graphs covering 9 ops with 0 graph breaks (0 unique) 2024-08-20T22:47:38.2826426Z accuracy pass_rate=51.28% 2024-08-20T22:47:38.2834712Z calls_captured gmean=0.00x mean=162.897x 2024-08-20T22:47:38.2839962Z unique_graphs gmean=0.00x mean=0.590x 2024-08-20T22:47:38.2844942Z graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:47:38.2849792Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:47:38.2855304Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T22:47:38.2860283Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T22:47:38.2864337Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T22:47:39.7290948Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_aot_inductor_torchbench_freezing_inference.csv 2024-08-20T22:47:40.0646562Z BERT_pytorch XFAIL 2024-08-20T22:47:40.0647110Z Background_Matting XFAIL 2024-08-20T22:47:40.0651498Z LearningToPaint PASS 2024-08-20T22:47:40.0656761Z Super_SloMo PASS 2024-08-20T22:47:40.0661900Z alexnet PASS 2024-08-20T22:47:40.0666726Z basic_gnn_edgecnn PASS 2024-08-20T22:47:40.0671792Z basic_gnn_gcn PASS 2024-08-20T22:47:40.0676728Z basic_gnn_gin PASS 2024-08-20T22:47:40.0681653Z basic_gnn_sage XFAIL 2024-08-20T22:47:40.0686565Z dcgan PASS 2024-08-20T22:47:40.0691449Z demucs PASS 2024-08-20T22:47:40.0696325Z densenet121 PASS 2024-08-20T22:47:40.0701496Z detectron2_fasterrcnn_r_101_c4 XFAIL 2024-08-20T22:47:40.0706363Z detectron2_fasterrcnn_r_101_dc5 XFAIL 2024-08-20T22:47:40.0711228Z detectron2_fasterrcnn_r_101_fpn XFAIL 2024-08-20T22:47:40.0716127Z detectron2_fasterrcnn_r_50_c4 XFAIL 2024-08-20T22:47:40.0720991Z detectron2_fasterrcnn_r_50_dc5 XFAIL 2024-08-20T22:47:40.0726134Z detectron2_fasterrcnn_r_50_fpn XFAIL 2024-08-20T22:47:40.0730714Z detectron2_maskrcnn_r_101_c4 XFAIL 2024-08-20T22:47:40.0735510Z detectron2_maskrcnn_r_101_fpn XFAIL 2024-08-20T22:47:40.0740592Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-08-20T22:47:40.0745475Z detectron2_maskrcnn_r_50_fpn XFAIL 2024-08-20T22:47:40.0750332Z dlrm XFAIL 2024-08-20T22:47:40.0755907Z doctr_det_predictor XFAIL 2024-08-20T22:47:40.0760926Z doctr_reco_predictor XFAIL 2024-08-20T22:47:40.0765780Z drq XFAIL 2024-08-20T22:47:40.0770658Z functorch_dp_cifar10 PASS 2024-08-20T22:47:40.0775494Z functorch_maml_omniglot PASS 2024-08-20T22:47:40.0780600Z hf_Albert PASS 2024-08-20T22:47:40.0785460Z hf_Bart PASS 2024-08-20T22:47:40.0790352Z hf_Bert PASS 2024-08-20T22:47:40.0795182Z hf_Bert_large PASS 2024-08-20T22:47:40.0800049Z hf_DistilBert PASS 2024-08-20T22:47:40.0804886Z hf_GPT2 PASS 2024-08-20T22:47:40.0809797Z hf_GPT2_large XFAIL 2024-08-20T22:47:40.0814852Z hf_T5_base PASS 2024-08-20T22:47:40.0819808Z hf_T5_large XFAIL 2024-08-20T22:47:40.0824659Z hf_distil_whisper PASS 2024-08-20T22:47:40.0829533Z lennard_jones PASS 2024-08-20T22:47:40.1210824Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_aot_inductor_torchbench_freezing_inference.csv 2024-08-20T22:47:40.4489562Z BERT_pytorch PASS 2024-08-20T22:47:40.4490098Z Background_Matting PASS 2024-08-20T22:47:40.4494545Z LearningToPaint PASS 2024-08-20T22:47:40.4499785Z Super_SloMo PASS 2024-08-20T22:47:40.4504660Z alexnet PASS 2024-08-20T22:47:40.4509643Z basic_gnn_edgecnn PASS 2024-08-20T22:47:40.4514621Z basic_gnn_gcn PASS 2024-08-20T22:47:40.4519621Z basic_gnn_gin PASS 2024-08-20T22:47:40.4524659Z basic_gnn_sage PASS 2024-08-20T22:47:40.4529653Z dcgan PASS 2024-08-20T22:47:40.4534623Z demucs PASS 2024-08-20T22:47:40.4539840Z densenet121 PASS 2024-08-20T22:47:40.4544831Z detectron2_fasterrcnn_r_101_c4 PASS 2024-08-20T22:47:40.4549785Z detectron2_fasterrcnn_r_101_dc5 PASS 2024-08-20T22:47:40.4555085Z detectron2_fasterrcnn_r_101_fpn PASS 2024-08-20T22:47:40.4560152Z detectron2_fasterrcnn_r_50_c4 PASS 2024-08-20T22:47:40.4565101Z detectron2_fasterrcnn_r_50_dc5 PASS 2024-08-20T22:47:40.4570035Z detectron2_fasterrcnn_r_50_fpn PASS 2024-08-20T22:47:40.4575022Z detectron2_maskrcnn_r_101_c4 PASS 2024-08-20T22:47:40.4580151Z detectron2_maskrcnn_r_101_fpn PASS 2024-08-20T22:47:40.4585193Z detectron2_maskrcnn_r_50_c4 PASS 2024-08-20T22:47:40.4590160Z detectron2_maskrcnn_r_50_fpn PASS 2024-08-20T22:47:40.4595106Z dlrm PASS 2024-08-20T22:47:40.4600071Z doctr_det_predictor PASS 2024-08-20T22:47:40.4605040Z doctr_reco_predictor PASS 2024-08-20T22:47:40.4609970Z drq PASS 2024-08-20T22:47:40.4614931Z functorch_dp_cifar10 PASS 2024-08-20T22:47:40.4620074Z functorch_maml_omniglot PASS 2024-08-20T22:47:40.4625088Z hf_Albert PASS 2024-08-20T22:47:40.4630024Z hf_Bart PASS 2024-08-20T22:47:40.4634953Z hf_Bert PASS 2024-08-20T22:47:40.4639896Z hf_Bert_large PASS 2024-08-20T22:47:40.4644889Z hf_DistilBert PASS 2024-08-20T22:47:40.4650064Z hf_GPT2 PASS 2024-08-20T22:47:40.4655111Z hf_GPT2_large PASS 2024-08-20T22:47:40.4660185Z hf_T5_base PASS 2024-08-20T22:47:40.4665117Z hf_T5_large PASS 2024-08-20T22:47:40.4670073Z hf_distil_whisper PASS 2024-08-20T22:47:40.4674957Z lennard_jones PASS 2024-08-20T22:47:40.5065184Z + cleanup_workspace 2024-08-20T22:47:40.5066289Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:47:40.5067418Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:47:40.5068381Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:47:40.5069021Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:47:40.5069856Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:47:40.5070789Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:47:40.5071498Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:47:41.4331117Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:47:41.4331626Z cat test/**/*_toprint.log || true 2024-08-20T22:47:41.4341069Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.4341565Z env: 2024-08-20T22:47:41.4341813Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.4342448Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.4343122Z ##[endgroup] 2024-08-20T22:47:41.4416161Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:47:41.4445030Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:47:41.4445477Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:47:41.4450995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.4451457Z env: 2024-08-20T22:47:41.4451694Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.4452282Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.4452947Z MONITOR_SCRIPT_PID: 117965 2024-08-20T22:47:41.4453255Z ##[endgroup] 2024-08-20T22:47:41.4658937Z Prepare all required actions 2024-08-20T22:47:41.4659431Z Getting action download info 2024-08-20T22:47:41.6817426Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:47:41.8566790Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:47:41.8567444Z with: 2024-08-20T22:47:41.8567980Z file-suffix: test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704 2024-08-20T22:47:41.8568648Z s3-bucket: gha-artifacts 2024-08-20T22:47:41.8568947Z env: 2024-08-20T22:47:41.8569171Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.8569745Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.8570367Z ##[endgroup] 2024-08-20T22:47:41.8594256Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:47:41.8594852Z # Remove any previous test jsons if they exist 2024-08-20T22:47:41.8595404Z rm -f test-jsons-*.zip 2024-08-20T22:47:41.8595881Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:47:41.8601749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.8602199Z env: 2024-08-20T22:47:41.8602436Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.8603005Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.8603915Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704 2024-08-20T22:47:41.8604551Z ##[endgroup] 2024-08-20T22:47:41.8833303Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:47:41.8873316Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:47:41.8876767Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:47:41.8888731Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:47:41.8902546Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:47:41.8926633Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:47:41.8927254Z # Remove any previous test reports if they exist 2024-08-20T22:47:41.8927760Z rm -f test-reports-*.zip 2024-08-20T22:47:41.8928322Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:47:41.8933566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.8934014Z env: 2024-08-20T22:47:41.8934251Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.8934830Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.8935729Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704 2024-08-20T22:47:41.8936365Z ##[endgroup] 2024-08-20T22:47:41.9119688Z adding: test/test-reports/inference_torchbench.csv (deflated 74%) 2024-08-20T22:47:41.9145042Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:47:41.9145667Z # Remove any previous usage logs if they exist 2024-08-20T22:47:41.9146151Z rm -f logs-*.zip 2024-08-20T22:47:41.9146809Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:47:41.9147590Z # so check to see if the file exists first 2024-08-20T22:47:41.9148064Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:47:41.9148572Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:47:41.9149045Z fi 2024-08-20T22:47:41.9149386Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:47:41.9149934Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:47:41.9150414Z fi 2024-08-20T22:47:41.9155652Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.9156140Z env: 2024-08-20T22:47:41.9156382Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.9157162Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.9158149Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704 2024-08-20T22:47:41.9158843Z ##[endgroup] 2024-08-20T22:47:41.9227111Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:47:41.9294229Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:47:41.9294924Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:47:41.9295450Z rm -f debug-*.zip 2024-08-20T22:47:41.9295803Z if [ -d 'test/debug' ]; then 2024-08-20T22:47:41.9296356Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:47:41.9296780Z fi 2024-08-20T22:47:41.9301836Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:41.9302328Z env: 2024-08-20T22:47:41.9302576Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.9303201Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.9304202Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704 2024-08-20T22:47:41.9304890Z ##[endgroup] 2024-08-20T22:47:41.9385445Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:41.9385860Z with: 2024-08-20T22:47:41.9386123Z s3-bucket: gha-artifacts 2024-08-20T22:47:41.9386546Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:41.9387007Z retention-days: 14 2024-08-20T22:47:41.9387312Z if-no-files-found: warn 2024-08-20T22:47:41.9387651Z path: test-jsons-*.zip 2024-08-20T22:47:41.9387974Z name: artifact 2024-08-20T22:47:41.9388244Z region: us-east-1 2024-08-20T22:47:41.9388525Z env: 2024-08-20T22:47:41.9388776Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:41.9389396Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:41.9390059Z ##[endgroup] 2024-08-20T22:47:42.2922012Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:42.2923418Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:42.2924094Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:42.2961985Z Starting upload of test-jsons-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:42.4095267Z Finished upload of test-jsons-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:42.4285193Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:42.4285608Z with: 2024-08-20T22:47:42.4285869Z s3-bucket: gha-artifacts 2024-08-20T22:47:42.4286282Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:42.4286729Z retention-days: 14 2024-08-20T22:47:42.4287029Z if-no-files-found: error 2024-08-20T22:47:42.4287484Z path: test-reports-*.zip 2024-08-20T22:47:42.4287793Z name: artifact 2024-08-20T22:47:42.4288046Z region: us-east-1 2024-08-20T22:47:42.4288467Z env: 2024-08-20T22:47:42.4288710Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:42.4289299Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:42.4289912Z ##[endgroup] 2024-08-20T22:47:42.7589174Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:42.7589791Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:42.7590397Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:42.7628149Z Starting upload of test-reports-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:42.8575851Z Finished upload of test-reports-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:42.8767646Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:42.8768036Z with: 2024-08-20T22:47:42.8768281Z s3-bucket: gha-artifacts 2024-08-20T22:47:42.8768676Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:42.8769117Z retention-days: 14 2024-08-20T22:47:42.8769555Z if-no-files-found: ignore 2024-08-20T22:47:42.8769883Z path: logs-*.zip 2024-08-20T22:47:42.8770156Z name: artifact 2024-08-20T22:47:42.8770409Z region: us-east-1 2024-08-20T22:47:42.8770672Z env: 2024-08-20T22:47:42.8770912Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:42.8771491Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:42.8772104Z ##[endgroup] 2024-08-20T22:47:43.2035422Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:43.2036035Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:43.2036651Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:43.2074896Z Starting upload of logs-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:43.3103840Z Finished upload of logs-test-cpu_aot_inductor_torchbench_freezing-1-2-amz2023.linux.12xlarge_29025550704.zip 2024-08-20T22:47:43.3315862Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:43.3316267Z with: 2024-08-20T22:47:43.3316499Z s3-bucket: gha-artifacts 2024-08-20T22:47:43.3316896Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:47:43.3317328Z retention-days: 14 2024-08-20T22:47:43.3317628Z if-no-files-found: ignore 2024-08-20T22:47:43.3317936Z path: debug-*.zip 2024-08-20T22:47:43.3318209Z name: artifact 2024-08-20T22:47:43.3318473Z region: us-east-1 2024-08-20T22:47:43.3318724Z env: 2024-08-20T22:47:43.3318962Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:43.3319542Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:43.3320159Z ##[endgroup] 2024-08-20T22:47:43.6540379Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:47:43.6722257Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:47:43.6722689Z # shellcheck disable=SC2156 2024-08-20T22:47:43.6723446Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:47:43.6729883Z shell: /usr/bin/bash -e {0} 2024-08-20T22:47:43.6730199Z env: 2024-08-20T22:47:43.6730444Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:43.6731030Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:43.6731647Z ##[endgroup] 2024-08-20T22:47:43.8897002Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:47:43.8897510Z with: 2024-08-20T22:47:43.8897735Z env: 2024-08-20T22:47:43.8897977Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:43.8898641Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:43.8899522Z ##[endgroup] 2024-08-20T22:47:43.8915839Z ##[group]Run set -eou pipefail 2024-08-20T22:47:43.8916200Z set -eou pipefail 2024-08-20T22:47:43.8916509Z  2024-08-20T22:47:43.8917101Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:47:43.8917683Z for _ in $(seq 1440); do 2024-08-20T22:47:43.8918105Z  # Break if no ssh session exists anymore 2024-08-20T22:47:43.8918549Z  if [ "$(who)" = "" ]; then 2024-08-20T22:47:43.8918910Z  break 2024-08-20T22:47:43.8919170Z  fi 2024-08-20T22:47:43.8919470Z  echo "." 2024-08-20T22:47:43.8919745Z  sleep 5 2024-08-20T22:47:43.8920020Z done 2024-08-20T22:47:43.8925119Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:43.8925573Z env: 2024-08-20T22:47:43.8925814Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:43.8926393Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:43.8927005Z ##[endgroup] 2024-08-20T22:47:43.8950182Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:47:43.9020255Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:47:43.9021024Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:47:43.9021612Z # shellcheck disable=SC2046 2024-08-20T22:47:43.9022061Z docker stop $(docker ps -q) || true 2024-08-20T22:47:43.9022515Z # Prune all of the docker images 2024-08-20T22:47:43.9022949Z docker system prune -af 2024-08-20T22:47:43.9028160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:43.9028616Z env: 2024-08-20T22:47:43.9028847Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:43.9029425Z DOCKER_CONTAINER_ID: dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:43.9030051Z ##[endgroup] 2024-08-20T22:47:54.8643899Z dceb742940ef 2024-08-20T22:47:56.4372954Z Deleted Containers: 2024-08-20T22:47:56.4373573Z dceb742940ef92a3b2f588bf8afb9ab9c4fac034ce3ff9e78be9df1c6b9184d3 2024-08-20T22:47:56.4374057Z 2024-08-20T22:48:00.3243762Z Deleted Images: 2024-08-20T22:48:00.3245711Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:48:00.3247905Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks@sha256:ed18f711327455e269271a17448384275684348a472ba289f00904bd14501fa9 2024-08-20T22:48:00.3249463Z deleted: sha256:822356886a68936cce34917f5ffc60d864918fab7594bb08c26bf1816dd6110d 2024-08-20T22:48:00.3250310Z deleted: sha256:ae6b018f5c327102b48941d39cd8d6fa6151893715290829a76e4e73add3a418 2024-08-20T22:48:00.3251187Z deleted: sha256:79fa7dddbf1ed3af5e8f10972d07ccea774929efe8578a1cc6604a00c93c1e02 2024-08-20T22:48:00.3252057Z deleted: sha256:06b439d80ae875bb9b96d0a23c4619aec5b5fb2f35b5048022ebe6061e19f7e1 2024-08-20T22:48:00.3252919Z deleted: sha256:52d3d63b327ae120e60f76fe27965e31638eb8d556b5e5f56f4773bd940373cf 2024-08-20T22:48:00.3253766Z deleted: sha256:5df173ad6b0f1ba16287832b6e5237513d152e87e627e478fc36ed16159697c0 2024-08-20T22:48:00.3254872Z deleted: sha256:b6c13bf75251ea08468f4040dbba32848f72cd00eb8c3003ecc1e879f73738ae 2024-08-20T22:48:00.3255719Z deleted: sha256:f2f37b6c898fc6a8c74683cc50a156f56abe71f20310c72338bb0ec23c71e8d6 2024-08-20T22:48:00.3256590Z deleted: sha256:55832a6aecdb11a84ddbbd38efcac3b62dd61a2b7832a1080535fc5730e1e336 2024-08-20T22:48:00.3257807Z deleted: sha256:bd86c04ac5f8ba1940bc991dfbae9e17a80293b5cf5fe568c694f46d59863011 2024-08-20T22:48:00.3258792Z deleted: sha256:88a76e0add806e67ed7bfd8ad7f862a8683d087dd3557cfc2b787da5f4c1ae0f 2024-08-20T22:48:00.3259660Z deleted: sha256:37a2749f2e5a556bb0369b3db12c47e4b82b41064f430896babceb49d97d2f9f 2024-08-20T22:48:00.3260486Z deleted: sha256:ac6214a417e93e76048bee154136a3117823ef38824b38f2659c74c7d09b57f8 2024-08-20T22:48:00.3261322Z deleted: sha256:bd60508bf2fcb98670e5ba90896321868f6fb9c5a2c859203e47512542f45b99 2024-08-20T22:48:00.3262179Z deleted: sha256:3fe3c9441ff3190f5cf87c066d3b5ca03aa3bb68071ef86e127c271ebb1ea02e 2024-08-20T22:48:00.3263159Z deleted: sha256:f9b8978a3de27376176f0400dcb17127771f2a8123ff323474e055dbae6eaba5 2024-08-20T22:48:00.3263984Z deleted: sha256:825e6cc758ce8478a096de8d367123d4d0a7f51fbe595867d2f950f714ca908e 2024-08-20T22:48:00.3264839Z deleted: sha256:dde33dfb74ab43131553ba775cd8561874596ba2e3d1b810c7796beba3d4fd38 2024-08-20T22:48:00.3265700Z deleted: sha256:809173faaa7c1289371a0c5a0927a6050de23417cddba0746dea79b10b68dd71 2024-08-20T22:48:00.3266545Z deleted: sha256:f63fee6ae14582933c75117b8d3115041d0177792e510d8dab0baca1b2202771 2024-08-20T22:48:00.3267390Z deleted: sha256:9432a4a5f47ddff6148fd5f8ade43d3a5566c0da2fbcd904daa917dc21c4c73d 2024-08-20T22:48:00.3268270Z deleted: sha256:b9ff51a3d1dc73e8a3ab751fa1a512026a101eaedbea74cfe01a2876b5111502 2024-08-20T22:48:00.3269132Z deleted: sha256:70e4013b779cc78a919225cad0a55d38948b0620ea119eb80dbb377ce8687f7e 2024-08-20T22:48:00.3269975Z deleted: sha256:edcf423609137384d678c9f677cae55933016f3ee76cc2ee47d93aad1346c427 2024-08-20T22:48:00.3270820Z deleted: sha256:bb2f6a3359e809db17e8b77499bf0d94200af9defc470b3adb3e1c28a6f1c6c1 2024-08-20T22:48:00.3271669Z deleted: sha256:eb646c0b0638f760f7c1046a113d3646e57513d19390881f4f6775ec702fa265 2024-08-20T22:48:00.3272515Z deleted: sha256:71c5137ede0421f7f00b1e93ee7bd6287c8ff6445615d0e1d1edddff2240789c 2024-08-20T22:48:00.3273633Z deleted: sha256:d2a20794246965e241eaaa98883ecd02a39901618f06536532bb4d9ed2b80f63 2024-08-20T22:48:00.3274454Z deleted: sha256:a8fc5137968c5840b35ff322f4796356303c4d4f0ed5c86c70218e20045936ac 2024-08-20T22:48:00.3275294Z deleted: sha256:1bd367d8cc65cb8808ddf7665064a0523467c7947ead2e1aed5112e9465e2fa8 2024-08-20T22:48:00.3276145Z deleted: sha256:736b605a3af6c5f2f82b517f0c1c517701a78def165ad99f3fb034198818994d 2024-08-20T22:48:00.3279186Z deleted: sha256:f2766fd6c51e0f9c3fdeed91ff144d049ed99f1a2cdce5c3b36a9f694af46045 2024-08-20T22:48:00.3280059Z deleted: sha256:688dce500613b1366475010344298a9b4eca441474fb871cfc37287a6eefbbf4 2024-08-20T22:48:00.3280893Z deleted: sha256:08f9d176a447289b58f9c70cdc76546487fc2700b1e5a7b79c19eb4f8f7c4608 2024-08-20T22:48:00.3281750Z deleted: sha256:538a2fc18bbc7da30a73c02bc8bb61bd6800a9d501caeaec614a32c41d53438d 2024-08-20T22:48:00.3282618Z deleted: sha256:75d7a20f4c66e80cf12fdf6d68787ee75c33478fd7c3e30c4696702ef9932993 2024-08-20T22:48:00.3283465Z deleted: sha256:5f36a35d97f9c931857392a1a616a5af4da00dfabe321b21f7637761d680b575 2024-08-20T22:48:00.3284327Z deleted: sha256:172cb1246a46e9dde7d50405f7fb31fd83a3a3e6ff40c811cfe100bd034807f6 2024-08-20T22:48:00.3285174Z deleted: sha256:d621ca4d3a5a15f4d74797b18bd0f4c44f63288c18e753fc4281cec69d82ea21 2024-08-20T22:48:00.3286038Z deleted: sha256:8e5eec944591edb185a54d6b495949dcae1da0be3fec8d30c3463e9fcde1a34e 2024-08-20T22:48:00.3286903Z deleted: sha256:656151b8c659bb4dab88293de022523e0e478e654f6922eaf53efb36ffb26e7e 2024-08-20T22:48:00.3287748Z deleted: sha256:9e2c8ec2c392f561f1016a988a64620f377203ed895815f024856cdef934be6b 2024-08-20T22:48:00.3288581Z deleted: sha256:c42f2e17dc37198a178852ecf9d947775b76af99ebdc2aa929af4b03b3896b3b 2024-08-20T22:48:00.3289435Z deleted: sha256:f52f63d8f16c18cde14e5738947c6e08323452e8599f723a6e041ed1ab33565a 2024-08-20T22:48:00.3290285Z deleted: sha256:343dc9f9b3e3c25dc528bbe1f8b4258289e512ff85f4b9c22ea2154c9c9d8143 2024-08-20T22:48:00.3291239Z deleted: sha256:5efa5da92a100a8c36c51a732af9534a8023041baf9d55ecf831c1386467d65a 2024-08-20T22:48:00.3292167Z deleted: sha256:da5cd90133fe64eb015b47ab4515d89305598acefc5c097b8e189da64be2d476 2024-08-20T22:48:00.3293087Z deleted: sha256:691e04884c858a1ea0a97381a7b1036bdfc6f9a48471608f0a5db9da9935555f 2024-08-20T22:48:00.3293875Z deleted: sha256:64100fb171394a71add5752ea3f6c432bcba0ac31aca67d8443c3722864804e1 2024-08-20T22:48:00.3294669Z deleted: sha256:a03197ec2ed41935ef46795a87ffd3d80b526fce1edf28550177f579cfb5df64 2024-08-20T22:48:00.3295453Z deleted: sha256:7bd9c2008289af25e772b1fc7b8e6470b513a5f76ce25743b9ef1bbe43ace5f7 2024-08-20T22:48:00.3296238Z deleted: sha256:0064821cdad60c711279a5964f88bf82c7582c296d21a0d1d0e97c60a47c1fa3 2024-08-20T22:48:00.3297087Z deleted: sha256:5825eb9623523140ad1e3ad683f3a83ab39054adf0b0fb4e472c2a7c3e7545ff 2024-08-20T22:48:00.3297892Z deleted: sha256:ff2d63ec14fa462e2edcd48d8ca8b678d79c9981bda17880818db15b8790e54b 2024-08-20T22:48:00.3298961Z deleted: sha256:79a5278d4bd098095d2f18fcfa476e8d54112dae4ac3d6c4c955e7fd9ba550f4 2024-08-20T22:48:00.3299792Z deleted: sha256:808ef6924cb221e2093f74413950d96ff62707fc13a194d83a0422315fccccdc 2024-08-20T22:48:00.3300647Z deleted: sha256:91aa45c12c4226ea9961e3acd079df4454bf2a55cb1bb9a2918f37bec89947bf 2024-08-20T22:48:00.3301509Z deleted: sha256:1f26a67abbf110b4883c6dd39e035f74248d28916c4a48be34296a7a96d1319b 2024-08-20T22:48:00.3302360Z deleted: sha256:1165de789bc53191b61c2b04dc04cf9d3d6d78f3d704776bd5a276d2caac8d09 2024-08-20T22:48:00.3303227Z deleted: sha256:aff63be2ff8d27d7172fb6dbecff0bea191a85c2431f8de65c0f81d40212587b 2024-08-20T22:48:00.3304103Z deleted: sha256:7c4290bd5a95669ca4d362a8672cb96b80693cc40af13f1ce061e86f835766d1 2024-08-20T22:48:00.3305073Z deleted: sha256:46ad9cfd3aa9b54d845ebc37c2e4c0d5fda90577227a7962049b71c825acdafe 2024-08-20T22:48:00.3305883Z deleted: sha256:aad12f955a8ca5766f6ca5d1467a6d2b81be713ff7224e8ae8eb92b68acf69a2 2024-08-20T22:48:00.3306677Z deleted: sha256:f5a053a7d9bda27126802be2ba52d187c26e934b7cc582850d2aaadd90631a1e 2024-08-20T22:48:00.3307466Z deleted: sha256:cf0f3c2eb852401868a3f959f6176f5a6b42f17f16e950f132f03293c764d733 2024-08-20T22:48:00.3308221Z deleted: sha256:7929c61352934651542ff23122529355349b2153871117dcfbaa4b45020a158d 2024-08-20T22:48:00.3308999Z deleted: sha256:119390db59d077c75caa11c49fee6ab586d061d27de1deeeec8608f330624f21 2024-08-20T22:48:00.3309849Z deleted: sha256:3b7424223daa8c5790ef526174f3780d9b1b54baa45d080127bfc72e60139a2b 2024-08-20T22:48:00.3310823Z deleted: sha256:f8986b3dfad5395410b72709f00789a62ee60d1b0bc1b27583f4c17af34b43fc 2024-08-20T22:48:00.3311640Z deleted: sha256:03db7655e84b428b1399087a9ed34180c120808cd00d94924a46eeaef73c37b0 2024-08-20T22:48:00.3312460Z deleted: sha256:b4a81bb8166fcf8ffbf063817b3202900c20586ca4bf369b21822764ab966164 2024-08-20T22:48:00.3313296Z deleted: sha256:afc241c47ba97b4a4f054cd0e6a5d90428b2675d00b160727187b7feabb81b14 2024-08-20T22:48:00.3314118Z deleted: sha256:e04730e6f5fc53e139356efef9c2c84b7bf6087c5a89876deaf1be58b9b1029b 2024-08-20T22:48:00.3314960Z deleted: sha256:7b22c8a2bc4bb0051ddc3ea51c917e7562663e4dafe4f9ad4a3870f4ec84bd14 2024-08-20T22:48:00.3315799Z deleted: sha256:c54b28287358751c9e19bd4f52b1d920ec5f4a254f8883b9d093528c08bc8b3f 2024-08-20T22:48:00.3316625Z deleted: sha256:17303ef9705fd22b570529dc8413ccba1a59fab7f5fef103bfe72e01c53d976e 2024-08-20T22:48:00.3317532Z deleted: sha256:1a4e7403586e712f2caf32f8bec553dafd88f78612fba5bd0c54e4efb89b9c41 2024-08-20T22:48:00.3318324Z deleted: sha256:d2f2462c603417aa43b82ea6f370ed4319cd12b3bda370d5a7ada0845daf3aaa 2024-08-20T22:48:00.3319121Z deleted: sha256:973bcd7518ef8f8e81b3b895b1ff4fe6e8a20915387d1f127edefccdcc3d867d 2024-08-20T22:48:00.3319896Z deleted: sha256:ab913355e8147522ebc4f29618812090844aa39956668afc4b91926827c327b3 2024-08-20T22:48:00.3320644Z deleted: sha256:69527fc1954cb8778833d15c0f3562fda1cbd230b11e649a974752f7a726cbf9 2024-08-20T22:48:00.3321422Z deleted: sha256:1b2a52846c812d7caf91918531cb04230500b31c3f695e444c1885aad0777b71 2024-08-20T22:48:00.3322211Z deleted: sha256:40f793cdc6b0a26ec9af5ea7188ad0ef19799a234e53eb4b8f3f756694a1000e 2024-08-20T22:48:00.3323015Z deleted: sha256:39b098ef95425e2d1bdb44559eda25dfc7bd62af2fb6bb691b540741278acb95 2024-08-20T22:48:00.3323892Z deleted: sha256:bc116c6279a58082c4ef9dac27e9d4d76bfdd78c2f2fe4439663d3dd2af55b57 2024-08-20T22:48:00.3324681Z deleted: sha256:57046a92505d1125e1af1100aaeb0e36175039b7dd36e67e7b4c554997d81b07 2024-08-20T22:48:00.3325450Z deleted: sha256:e868a8284f40af2175a17211327660844c1ac66ee31919f52c5e4e2b9564213a 2024-08-20T22:48:00.3326237Z deleted: sha256:ba5f7b6add2ae68d2962f19c3d981adc39633683af6dc3392c354cce87b7e8f9 2024-08-20T22:48:00.3327008Z deleted: sha256:3a06302b80c87d1030f701f23861d185613a3a4269faf533bb1b82b092a82887 2024-08-20T22:48:00.3327852Z deleted: sha256:dbdee317045b3156074f52d1fb32d613a48b6d8ed13c3e400a9f2c7bcceaf808 2024-08-20T22:48:00.3328658Z deleted: sha256:9574f911b87c39bbbb08a0238b82fb53abdedad1f387beb999ce1d36f3845e98 2024-08-20T22:48:00.3329467Z deleted: sha256:15d49efcad797dff0aa23e3b025b50c1c961668637e4bd9dff2361eaf3405c74 2024-08-20T22:48:00.3330275Z deleted: sha256:1b9b7346fee7abbc7f5538eaa23548bd05a45abe8daf6794024be0c8ad7d60bb 2024-08-20T22:48:00.3330762Z 2024-08-20T22:48:00.3330894Z Total reclaimed space: 35.51GB 2024-08-20T22:48:00.3396368Z Post job cleanup. 2024-08-20T22:48:00.3443389Z Post job cleanup. 2024-08-20T22:48:00.4298115Z [command]/usr/bin/git version 2024-08-20T22:48:00.4360706Z git version 2.40.1 2024-08-20T22:48:00.4399530Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/47a29dac-756d-4284-8003-05536243cea6' before making global git config changes 2024-08-20T22:48:00.4400822Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:48:00.4402493Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:48:00.4438589Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:48:00.4471838Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:48:00.4786304Z Entering 'android/libs/fbjni' 2024-08-20T22:48:00.4838773Z Entering 'third_party/FP16' 2024-08-20T22:48:00.4890471Z Entering 'third_party/FXdiv' 2024-08-20T22:48:00.4942904Z Entering 'third_party/NNPACK' 2024-08-20T22:48:00.4995643Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:48:00.5046926Z Entering 'third_party/XNNPACK' 2024-08-20T22:48:00.5114767Z Entering 'third_party/benchmark' 2024-08-20T22:48:00.5164645Z Entering 'third_party/cpp-httplib' 2024-08-20T22:48:00.5217016Z Entering 'third_party/cpuinfo' 2024-08-20T22:48:00.5269904Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:48:00.5322603Z Entering 'third_party/cutlass' 2024-08-20T22:48:00.5380624Z Entering 'third_party/eigen' 2024-08-20T22:48:00.5436022Z Entering 'third_party/fbgemm' 2024-08-20T22:48:00.5487938Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:48:00.5536841Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:48:00.5590177Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:48:00.5647528Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:48:00.5700225Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:48:00.5749125Z Entering 'third_party/flatbuffers' 2024-08-20T22:48:00.5804998Z Entering 'third_party/fmt' 2024-08-20T22:48:00.5856100Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:48:00.5906243Z Entering 'third_party/gloo' 2024-08-20T22:48:00.5956369Z Entering 'third_party/googletest' 2024-08-20T22:48:00.6008210Z Entering 'third_party/ideep' 2024-08-20T22:48:00.6055948Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:48:00.6113509Z Entering 'third_party/ittapi' 2024-08-20T22:48:00.6161950Z Entering 'third_party/kineto' 2024-08-20T22:48:00.6213771Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:48:00.6261562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:48:00.6322345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:48:00.6373761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:48:00.6421720Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:48:00.6470091Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:48:00.6524192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:48:00.6574811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:48:00.6623838Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:48:00.6675988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:48:00.6727166Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:48:00.6776440Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:48:00.6825767Z Entering 'third_party/mimalloc' 2024-08-20T22:48:00.6877514Z Entering 'third_party/nccl/nccl' 2024-08-20T22:48:00.6928620Z Entering 'third_party/nlohmann' 2024-08-20T22:48:00.6980273Z Entering 'third_party/onnx' 2024-08-20T22:48:00.7044834Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:48:00.7100118Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:48:00.7150826Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:48:00.7206284Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:48:00.7256506Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:48:00.7309310Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:48:00.7357762Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:48:00.7410077Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:48:00.7458032Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:48:00.7514638Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:48:00.7561508Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:48:00.7614010Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:48:00.7665550Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:48:00.7737292Z Entering 'third_party/pocketfft' 2024-08-20T22:48:00.7786239Z Entering 'third_party/protobuf' 2024-08-20T22:48:00.7839222Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:48:00.7889380Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:48:00.7942949Z Entering 'third_party/psimd' 2024-08-20T22:48:00.7995147Z Entering 'third_party/pthreadpool' 2024-08-20T22:48:00.8044438Z Entering 'third_party/pybind11' 2024-08-20T22:48:00.8095089Z Entering 'third_party/python-peachpy' 2024-08-20T22:48:00.8143804Z Entering 'third_party/sleef' 2024-08-20T22:48:00.8196528Z Entering 'third_party/tensorpipe' 2024-08-20T22:48:00.8246013Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:48:00.8294815Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:48:00.8348617Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:48:00.8398886Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:48:00.8449321Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:48:00.8516635Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:48:00.8543227Z http.https://github.com/.extraheader 2024-08-20T22:48:00.8551059Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:48:00.8583395Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:48:00.8865330Z Entering 'android/libs/fbjni' 2024-08-20T22:48:00.8900087Z http.https://github.com/.extraheader 2024-08-20T22:48:00.8933100Z Entering 'third_party/FP16' 2024-08-20T22:48:00.8989278Z http.https://github.com/.extraheader 2024-08-20T22:48:00.8998761Z Entering 'third_party/FXdiv' 2024-08-20T22:48:00.9035474Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9067335Z Entering 'third_party/NNPACK' 2024-08-20T22:48:00.9100614Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9133697Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:48:00.9165918Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9199573Z Entering 'third_party/XNNPACK' 2024-08-20T22:48:00.9233284Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9282565Z Entering 'third_party/benchmark' 2024-08-20T22:48:00.9316563Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9347462Z Entering 'third_party/cpp-httplib' 2024-08-20T22:48:00.9381285Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9418246Z Entering 'third_party/cpuinfo' 2024-08-20T22:48:00.9451072Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9483715Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:48:00.9517496Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9547639Z Entering 'third_party/cutlass' 2024-08-20T22:48:00.9581544Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9623389Z Entering 'third_party/eigen' 2024-08-20T22:48:00.9656717Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9694154Z Entering 'third_party/fbgemm' 2024-08-20T22:48:00.9727798Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9757689Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:48:00.9792047Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9829405Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:48:00.9863754Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9899762Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:48:00.9932494Z http.https://github.com/.extraheader 2024-08-20T22:48:00.9973570Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:48:01.0006643Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0040497Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:48:01.0074702Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0106720Z Entering 'third_party/flatbuffers' 2024-08-20T22:48:01.0140150Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0176349Z Entering 'third_party/fmt' 2024-08-20T22:48:01.0209990Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0241705Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:48:01.0276810Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0308313Z Entering 'third_party/gloo' 2024-08-20T22:48:01.0341263Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0374179Z Entering 'third_party/googletest' 2024-08-20T22:48:01.0407410Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0438770Z Entering 'third_party/ideep' 2024-08-20T22:48:01.0473367Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0502540Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:48:01.0535765Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0577284Z Entering 'third_party/ittapi' 2024-08-20T22:48:01.0611961Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0644336Z Entering 'third_party/kineto' 2024-08-20T22:48:01.0679147Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0710603Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:48:01.0742499Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0774592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:48:01.0807703Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0840925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:48:01.0875691Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0911105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:48:01.0945054Z http.https://github.com/.extraheader 2024-08-20T22:48:01.0978556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:48:01.1011280Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1042706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:48:01.1077515Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1110259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:48:01.1142819Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1175794Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:48:01.1209899Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1242057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:48:01.1276445Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1315953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:48:01.1351415Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1382979Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:48:01.1417625Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1457197Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:48:01.1491760Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1525111Z Entering 'third_party/mimalloc' 2024-08-20T22:48:01.1556805Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1594795Z Entering 'third_party/nccl/nccl' 2024-08-20T22:48:01.1626294Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1662068Z Entering 'third_party/nlohmann' 2024-08-20T22:48:01.1696246Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1728764Z Entering 'third_party/onnx' 2024-08-20T22:48:01.1760896Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1809015Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:48:01.1842396Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1873889Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:48:01.1908132Z http.https://github.com/.extraheader 2024-08-20T22:48:01.1944462Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:48:01.1979515Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2012225Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:48:01.2044499Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2077657Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:48:01.2109528Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2142191Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:48:01.2176451Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2207628Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:48:01.2240918Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2273958Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:48:01.2305851Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2340820Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:48:01.2375427Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2405532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:48:01.2438057Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2475036Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:48:01.2507226Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2544122Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:48:01.2577638Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2612512Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:48:01.2646552Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2699553Z Entering 'third_party/pocketfft' 2024-08-20T22:48:01.2731862Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2761669Z Entering 'third_party/protobuf' 2024-08-20T22:48:01.2796708Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2836070Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:48:01.2869955Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2903268Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:48:01.2936595Z http.https://github.com/.extraheader 2024-08-20T22:48:01.2976846Z Entering 'third_party/psimd' 2024-08-20T22:48:01.3010034Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3041121Z Entering 'third_party/pthreadpool' 2024-08-20T22:48:01.3075761Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3106127Z Entering 'third_party/pybind11' 2024-08-20T22:48:01.3140151Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3176595Z Entering 'third_party/python-peachpy' 2024-08-20T22:48:01.3209562Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3240666Z Entering 'third_party/sleef' 2024-08-20T22:48:01.3275143Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3305428Z Entering 'third_party/tensorpipe' 2024-08-20T22:48:01.3339883Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3372480Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:48:01.3406049Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3437695Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:48:01.3469208Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3501605Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:48:01.3535664Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3570661Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:48:01.3602541Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3634318Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:48:01.3666795Z http.https://github.com/.extraheader 2024-08-20T22:48:01.3791407Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:48:01.3807124Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:48:01.3812158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:01.3812757Z ##[endgroup] 2024-08-20T22:48:07.6236049Z Cleaning up orphan processes