2024-08-20T21:40:05.9797342Z Current runner version: '2.319.1' 2024-08-20T21:40:05.9803257Z Runner name: 'i-0315eba2eab835699' 2024-08-20T21:40:05.9804046Z Runner group name: 'Default' 2024-08-20T21:40:05.9804787Z Machine name: 'ip-10-0-17-252' 2024-08-20T21:40:05.9820933Z Testing runner upgrade compatibility 2024-08-20T21:40:06.1845107Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T21:40:06.1847649Z Actions: read 2024-08-20T21:40:06.1848133Z Attestations: read 2024-08-20T21:40:06.1848543Z Checks: read 2024-08-20T21:40:06.1848991Z Contents: read 2024-08-20T21:40:06.1849435Z Deployments: read 2024-08-20T21:40:06.1849884Z Discussions: read 2024-08-20T21:40:06.1850321Z Issues: read 2024-08-20T21:40:06.1850740Z Metadata: read 2024-08-20T21:40:06.1851169Z Packages: read 2024-08-20T21:40:06.1851726Z Pages: read 2024-08-20T21:40:06.1852145Z PullRequests: read 2024-08-20T21:40:06.1852648Z RepositoryProjects: read 2024-08-20T21:40:06.1853211Z SecurityEvents: read 2024-08-20T21:40:06.1853639Z Statuses: read 2024-08-20T21:40:06.1854308Z ##[endgroup] 2024-08-20T21:40:06.1857550Z Secret source: Actions 2024-08-20T21:40:06.1858189Z Prepare workflow directory 2024-08-20T21:40:06.5220749Z Prepare all required actions 2024-08-20T21:40:06.5381565Z Getting action download info 2024-08-20T21:40:06.7207575Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T21:40:07.0568914Z Download action repository 'pytorch/pytorch@main' (SHA:15b5a0b67fc3f34fb0bf1afa6f91e0c4c2b7fd8d) 2024-08-20T21:40:09.9703815Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T21:40:10.1807495Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T21:40:10.4369537Z Getting action download info 2024-08-20T21:40:10.5334458Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T21:40:10.7181912Z Getting action download info 2024-08-20T21:40:10.8206847Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T21:40:11.0082457Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T21:40:11.0084633Z ##[group] Inputs 2024-08-20T21:40:11.0085039Z build-environment: linux-jammy-py3.8-gcc11-build 2024-08-20T21:40:11.0105281Z test-matrix: {"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]} 2024-08-20T21:40:11.0124943Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:40:11.0126018Z sync-tag: 2024-08-20T21:40:11.0127037Z timeout-minutes: 240 2024-08-20T21:40:11.0127347Z use-gha: 2024-08-20T21:40:11.0127591Z dashboard-tag: 2024-08-20T21:40:11.0127879Z s3-bucket: gha-artifacts 2024-08-20T21:40:11.0128204Z aws-role-to-assume: 2024-08-20T21:40:11.0128508Z ##[endgroup] 2024-08-20T21:40:11.0129498Z Complete job name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:40:11.0892337Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T21:40:11.1030543Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T21:40:11.1039215Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:40:11.1039929Z ##[endgroup] 2024-08-20T21:40:12.6922980Z Runner Type: amz2023.linux.12xlarge 2024-08-20T21:40:12.6923485Z Instance Type: c5.12xlarge 2024-08-20T21:40:12.6924140Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T21:40:12.6924676Z AMI ID: ami-06c68f701d8090592 2024-08-20T21:40:18.0223265Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T21:40:18.0223969Z with: 2024-08-20T21:40:18.0224605Z github-secret: *** 2024-08-20T21:40:18.0225492Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:40:18.0226456Z activate-with-label: false 2024-08-20T21:40:18.0226769Z label: with-ssh 2024-08-20T21:40:18.0227057Z remove-existing-keys: true 2024-08-20T21:40:18.0227383Z fail-silently: true 2024-08-20T21:40:18.0227663Z env: 2024-08-20T21:40:18.0227891Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.0228196Z ##[endgroup] 2024-08-20T21:40:18.1033039Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T21:40:18.1034604Z ciflow reference detected, attempting to extract PR number 2024-08-20T21:40:18.3809359Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T21:40:18.4542568Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T21:40:18.4543553Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T21:40:18.5905793Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T21:40:18.5919053Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T21:40:18.5943848Z Login using: ssh ec2-user@ec2-44-192-63-246.compute-1.amazonaws.com 2024-08-20T21:40:18.5944692Z All testing is done inside the container, to start an interactive session run: 2024-08-20T21:40:18.5945854Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:40:18.6067442Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T21:40:18.6067990Z with: 2024-08-20T21:40:18.6068247Z submodules: recursive 2024-08-20T21:40:18.6068540Z fetch-depth: 0 2024-08-20T21:40:18.6068810Z env: 2024-08-20T21:40:18.6069061Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.6069362Z ##[endgroup] 2024-08-20T21:40:18.6253901Z ##[group]Run retry () { 2024-08-20T21:40:18.6254280Z retry () { 2024-08-20T21:40:18.6254781Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T21:40:18.6255346Z } 2024-08-20T21:40:18.6255639Z echo "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.6256057Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T21:40:18.6256532Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.6256983Z else 2024-08-20T21:40:18.6257306Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.6257733Z fi 2024-08-20T21:40:18.6258085Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T21:40:18.6266402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:40:18.6266892Z env: 2024-08-20T21:40:18.6267146Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:18.6267449Z NO_SUDO: 2024-08-20T21:40:18.6267706Z ##[endgroup] 2024-08-20T21:40:18.6294827Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.5589979Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T21:40:21.5590371Z with: 2024-08-20T21:40:21.5590666Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:40:21.5591075Z fetch-depth: 0 2024-08-20T21:40:21.5591353Z submodules: recursive 2024-08-20T21:40:21.5591646Z quiet-checkout: true 2024-08-20T21:40:21.5591958Z repository: pytorch/pytorch 2024-08-20T21:40:21.5592403Z token: *** 2024-08-20T21:40:21.5592657Z ssh-strict: true 2024-08-20T21:40:21.5592936Z persist-credentials: true 2024-08-20T21:40:21.5593258Z clean: true 2024-08-20T21:40:21.5593561Z sparse-checkout-cone-mode: true 2024-08-20T21:40:21.5594088Z lfs: false 2024-08-20T21:40:21.5594341Z set-safe-directory: true 2024-08-20T21:40:21.5594645Z env: 2024-08-20T21:40:21.5594884Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:40:21.5595173Z ##[endgroup] 2024-08-20T21:40:21.6494200Z Syncing repository: pytorch/pytorch 2024-08-20T21:40:21.6495812Z ##[group]Getting Git version info 2024-08-20T21:40:21.6496580Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:40:21.6497434Z [command]/usr/bin/git version 2024-08-20T21:40:21.6497795Z git version 2.40.1 2024-08-20T21:40:21.6499160Z ##[endgroup] 2024-08-20T21:40:21.6510258Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/41ab57f3-d63c-42da-97b8-dca8ac2f4b53' before making global git config changes 2024-08-20T21:40:21.6511540Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T21:40:21.6512721Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.6543667Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:40:21.6548529Z ##[group]Initializing the repository 2024-08-20T21:40:21.6551116Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:40:21.6582477Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T21:40:21.6583357Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T21:40:21.6584169Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T21:40:21.6584745Z hint: 2024-08-20T21:40:21.6585329Z hint: git config --global init.defaultBranch 2024-08-20T21:40:21.6585783Z hint: 2024-08-20T21:40:21.6586303Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T21:40:21.6587162Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T21:40:21.6588007Z hint: 2024-08-20T21:40:21.6588313Z hint: git branch -m 2024-08-20T21:40:21.6589081Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T21:40:21.6591221Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T21:40:21.6621874Z ##[endgroup] 2024-08-20T21:40:21.6622434Z ##[group]Disabling automatic garbage collection 2024-08-20T21:40:21.6624432Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T21:40:21.6653920Z ##[endgroup] 2024-08-20T21:40:21.6654428Z ##[group]Setting up auth 2024-08-20T21:40:21.6659538Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T21:40:21.6689693Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T21:40:21.6974497Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T21:40:21.7004439Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T21:40:21.7266244Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:40:21.7314271Z ##[endgroup] 2024-08-20T21:40:21.7314820Z ##[group]Fetching the repository 2024-08-20T21:40:21.7318756Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T21:40:24.2366807Z remote: Enumerating objects: 1008453 2024-08-20T21:40:24.2368138Z remote: Enumerating objects: 1010152, done. 2024-08-20T21:40:24.2368757Z remote: Counting objects: 0% (1/1699) 2024-08-20T21:40:24.2369272Z remote: Counting objects: 1% (17/1699) 2024-08-20T21:40:24.2369809Z remote: Counting objects: 2% (34/1699) 2024-08-20T21:40:24.2370624Z remote: Counting objects: 3% (51/1699) 2024-08-20T21:40:24.2371170Z remote: Counting objects: 4% (68/1699) 2024-08-20T21:40:24.2371841Z remote: Counting objects: 5% (85/1699) 2024-08-20T21:40:24.2372350Z remote: Counting objects: 6% (102/1699) 2024-08-20T21:40:24.2372846Z remote: Counting objects: 7% (119/1699) 2024-08-20T21:40:24.2373358Z remote: Counting objects: 8% (136/1699) 2024-08-20T21:40:24.2373869Z remote: Counting objects: 9% (153/1699) 2024-08-20T21:40:24.2374373Z remote: Counting objects: 10% (170/1699) 2024-08-20T21:40:24.2374856Z remote: Counting objects: 11% (187/1699) 2024-08-20T21:40:24.2375365Z remote: Counting objects: 12% (204/1699) 2024-08-20T21:40:24.2375876Z remote: Counting objects: 13% (221/1699) 2024-08-20T21:40:24.2376385Z remote: Counting objects: 14% (238/1699) 2024-08-20T21:40:24.2376883Z remote: Counting objects: 15% (255/1699) 2024-08-20T21:40:24.2377394Z remote: Counting objects: 16% (272/1699) 2024-08-20T21:40:24.2377897Z remote: Counting objects: 17% (289/1699) 2024-08-20T21:40:24.2378399Z remote: Counting objects: 18% (306/1699) 2024-08-20T21:40:24.2378880Z remote: Counting objects: 19% (323/1699) 2024-08-20T21:40:24.2379388Z remote: Counting objects: 20% (340/1699) 2024-08-20T21:40:24.2379890Z remote: Counting objects: 21% (357/1699) 2024-08-20T21:40:24.2380390Z remote: Counting objects: 22% (374/1699) 2024-08-20T21:40:24.2380871Z remote: Counting objects: 23% (391/1699) 2024-08-20T21:40:24.2381373Z remote: Counting objects: 24% (408/1699) 2024-08-20T21:40:24.2381874Z remote: Counting objects: 25% (425/1699) 2024-08-20T21:40:24.2382370Z remote: Counting objects: 26% (442/1699) 2024-08-20T21:40:24.2382854Z remote: Counting objects: 27% (459/1699) 2024-08-20T21:40:24.2383569Z remote: Counting objects: 28% (476/1699) 2024-08-20T21:40:24.2384098Z remote: Counting objects: 29% (493/1699) 2024-08-20T21:40:24.2384604Z remote: Counting objects: 30% (510/1699) 2024-08-20T21:40:24.2385092Z remote: Counting objects: 31% (527/1699) 2024-08-20T21:40:24.2385604Z remote: Counting objects: 32% (544/1699) 2024-08-20T21:40:24.2386274Z remote: Counting objects: 33% (561/1699) 2024-08-20T21:40:24.2386804Z remote: Counting objects: 34% (578/1699) 2024-08-20T21:40:24.2387305Z remote: Counting objects: 35% (595/1699) 2024-08-20T21:40:24.2387789Z remote: Counting objects: 36% (612/1699) 2024-08-20T21:40:24.2388290Z remote: Counting objects: 37% (629/1699) 2024-08-20T21:40:24.2388790Z remote: Counting objects: 38% (646/1699) 2024-08-20T21:40:24.2389294Z remote: Counting objects: 39% (663/1699) 2024-08-20T21:40:24.2389777Z remote: Counting objects: 40% (680/1699) 2024-08-20T21:40:24.2390281Z remote: Counting objects: 41% (697/1699) 2024-08-20T21:40:24.2390821Z remote: Counting objects: 42% (714/1699) 2024-08-20T21:40:24.2391318Z remote: Counting objects: 43% (731/1699) 2024-08-20T21:40:24.2391815Z remote: Counting objects: 44% (748/1699) 2024-08-20T21:40:24.2392293Z remote: Counting objects: 45% (765/1699) 2024-08-20T21:40:24.2392798Z remote: Counting objects: 46% (782/1699) 2024-08-20T21:40:24.2393299Z remote: Counting objects: 47% (799/1699) 2024-08-20T21:40:24.2393802Z remote: Counting objects: 48% (816/1699) 2024-08-20T21:40:24.2394285Z remote: Counting objects: 49% (833/1699) 2024-08-20T21:40:24.2394786Z remote: Counting objects: 50% (850/1699) 2024-08-20T21:40:24.2395286Z remote: Counting objects: 51% (867/1699) 2024-08-20T21:40:24.2395783Z remote: Counting objects: 52% (884/1699) 2024-08-20T21:40:24.2396264Z remote: Counting objects: 53% (901/1699) 2024-08-20T21:40:24.2396778Z remote: Counting objects: 54% (918/1699) 2024-08-20T21:40:24.2397372Z remote: Counting objects: 55% (935/1699) 2024-08-20T21:40:24.2397878Z remote: Counting objects: 56% (952/1699) 2024-08-20T21:40:24.2398364Z remote: Counting objects: 57% (969/1699) 2024-08-20T21:40:24.2398872Z remote: Counting objects: 58% (986/1699) 2024-08-20T21:40:24.2399384Z remote: Counting objects: 59% (1003/1699) 2024-08-20T21:40:24.2399912Z remote: Counting objects: 60% (1020/1699) 2024-08-20T21:40:24.2400422Z remote: Counting objects: 61% (1037/1699) 2024-08-20T21:40:24.2400951Z remote: Counting objects: 62% (1054/1699) 2024-08-20T21:40:24.2401470Z remote: Counting objects: 63% (1071/1699) 2024-08-20T21:40:24.2401989Z remote: Counting objects: 64% (1088/1699) 2024-08-20T21:40:24.2402492Z remote: Counting objects: 65% (1105/1699) 2024-08-20T21:40:24.2403016Z remote: Counting objects: 66% (1122/1699) 2024-08-20T21:40:24.2403534Z remote: Counting objects: 67% (1139/1699) 2024-08-20T21:40:24.2404058Z remote: Counting objects: 68% (1156/1699) 2024-08-20T21:40:24.2404565Z remote: Counting objects: 69% (1173/1699) 2024-08-20T21:40:24.2405086Z remote: Counting objects: 70% (1190/1699) 2024-08-20T21:40:24.2405619Z remote: Counting objects: 71% (1207/1699) 2024-08-20T21:40:24.2406150Z remote: Counting objects: 72% (1224/1699) 2024-08-20T21:40:24.2406656Z remote: Counting objects: 73% (1241/1699) 2024-08-20T21:40:24.2407179Z remote: Counting objects: 74% (1258/1699) 2024-08-20T21:40:24.2407706Z remote: Counting objects: 75% (1275/1699) 2024-08-20T21:40:24.2408216Z remote: Counting objects: 76% (1292/1699) 2024-08-20T21:40:24.2408751Z remote: Counting objects: 77% (1309/1699) 2024-08-20T21:40:24.2409280Z remote: Counting objects: 78% (1326/1699) 2024-08-20T21:40:24.2409881Z remote: Counting objects: 79% (1343/1699) 2024-08-20T21:40:24.2410399Z remote: Counting objects: 80% (1360/1699) 2024-08-20T21:40:24.2411191Z remote: Counting objects: 81% (1377/1699) 2024-08-20T21:40:24.2411959Z remote: Counting objects: 82% (1394/1699) 2024-08-20T21:40:24.2412580Z remote: Counting objects: 83% (1411/1699) 2024-08-20T21:40:24.2413262Z remote: Counting objects: 84% (1428/1699) 2024-08-20T21:40:24.2413828Z remote: Counting objects: 85% (1445/1699) 2024-08-20T21:40:24.2414450Z remote: Counting objects: 86% (1462/1699) 2024-08-20T21:40:24.2415107Z remote: Counting objects: 87% (1479/1699) 2024-08-20T21:40:24.2415722Z remote: Counting objects: 88% (1496/1699) 2024-08-20T21:40:24.2416288Z remote: Counting objects: 89% (1513/1699) 2024-08-20T21:40:24.2416965Z remote: Counting objects: 90% (1530/1699) 2024-08-20T21:40:24.2417581Z remote: Counting objects: 91% (1547/1699) 2024-08-20T21:40:24.2418254Z remote: Counting objects: 92% (1564/1699) 2024-08-20T21:40:24.2418887Z remote: Counting objects: 93% (1581/1699) 2024-08-20T21:40:24.2419457Z remote: Counting objects: 94% (1598/1699) 2024-08-20T21:40:24.2420117Z remote: Counting objects: 95% (1615/1699) 2024-08-20T21:40:24.2420736Z remote: Counting objects: 96% (1632/1699) 2024-08-20T21:40:24.2421352Z remote: Counting objects: 97% (1649/1699) 2024-08-20T21:40:24.2421959Z remote: Counting objects: 98% (1666/1699) 2024-08-20T21:40:24.2422575Z remote: Counting objects: 99% (1683/1699) 2024-08-20T21:40:24.2423273Z remote: Counting objects: 100% (1699/1699) 2024-08-20T21:40:24.2423968Z remote: Counting objects: 100% (1699/1699), done. 2024-08-20T21:40:24.2573081Z remote: Compressing objects: 0% (1/846) 2024-08-20T21:40:24.2886608Z remote: Compressing objects: 1% (9/846) 2024-08-20T21:40:24.3128054Z remote: Compressing objects: 2% (17/846) 2024-08-20T21:40:24.3649598Z remote: Compressing objects: 3% (26/846) 2024-08-20T21:40:24.4498060Z remote: Compressing objects: 4% (34/846) 2024-08-20T21:40:24.4962258Z remote: Compressing objects: 5% (43/846) 2024-08-20T21:40:24.5463220Z remote: Compressing objects: 6% (51/846) 2024-08-20T21:40:24.5841508Z remote: Compressing objects: 7% (60/846) 2024-08-20T21:40:24.6141356Z remote: Compressing objects: 8% (68/846) 2024-08-20T21:40:24.6432404Z remote: Compressing objects: 9% (77/846) 2024-08-20T21:40:24.6695845Z remote: Compressing objects: 10% (85/846) 2024-08-20T21:40:24.6880389Z remote: Compressing objects: 11% (94/846) 2024-08-20T21:40:24.7113568Z remote: Compressing objects: 12% (102/846) 2024-08-20T21:40:24.7279728Z remote: Compressing objects: 13% (110/846) 2024-08-20T21:40:24.7383736Z remote: Compressing objects: 14% (119/846) 2024-08-20T21:40:24.7466474Z remote: Compressing objects: 15% (127/846) 2024-08-20T21:40:24.7558243Z remote: Compressing objects: 16% (136/846) 2024-08-20T21:40:24.7642855Z remote: Compressing objects: 17% (144/846) 2024-08-20T21:40:24.7690787Z remote: Compressing objects: 18% (153/846) 2024-08-20T21:40:24.7738844Z remote: Compressing objects: 19% (161/846) 2024-08-20T21:40:24.7772272Z remote: Compressing objects: 20% (170/846) 2024-08-20T21:40:24.7803289Z remote: Compressing objects: 21% (178/846) 2024-08-20T21:40:24.7809465Z remote: Compressing objects: 22% (187/846) 2024-08-20T21:40:24.7810020Z remote: Compressing objects: 23% (195/846) 2024-08-20T21:40:24.7814067Z remote: Compressing objects: 24% (204/846) 2024-08-20T21:40:24.7817709Z remote: Compressing objects: 25% (212/846) 2024-08-20T21:40:24.7823119Z remote: Compressing objects: 26% (220/846) 2024-08-20T21:40:24.7834025Z remote: Compressing objects: 27% (229/846) 2024-08-20T21:40:24.7845547Z remote: Compressing objects: 28% (237/846) 2024-08-20T21:40:24.7861763Z remote: Compressing objects: 29% (246/846) 2024-08-20T21:40:24.7864763Z remote: Compressing objects: 30% (254/846) 2024-08-20T21:40:24.7875868Z remote: Compressing objects: 31% (263/846) 2024-08-20T21:40:24.7885333Z remote: Compressing objects: 32% (271/846) 2024-08-20T21:40:24.7891469Z remote: Compressing objects: 33% (280/846) 2024-08-20T21:40:24.7901125Z remote: Compressing objects: 34% (288/846) 2024-08-20T21:40:24.7906928Z remote: Compressing objects: 35% (297/846) 2024-08-20T21:40:24.7911633Z remote: Compressing objects: 36% (305/846) 2024-08-20T21:40:24.7921529Z remote: Compressing objects: 37% (314/846) 2024-08-20T21:40:24.7927836Z remote: Compressing objects: 38% (322/846) 2024-08-20T21:40:24.7935101Z remote: Compressing objects: 39% (330/846) 2024-08-20T21:40:24.7938668Z remote: Compressing objects: 40% (339/846) 2024-08-20T21:40:24.7944548Z remote: Compressing objects: 41% (347/846) 2024-08-20T21:40:24.7952656Z remote: Compressing objects: 42% (356/846) 2024-08-20T21:40:24.7957172Z remote: Compressing objects: 43% (364/846) 2024-08-20T21:40:24.7961532Z remote: Compressing objects: 44% (373/846) 2024-08-20T21:40:24.7968845Z remote: Compressing objects: 45% (381/846) 2024-08-20T21:40:24.7974291Z remote: Compressing objects: 46% (390/846) 2024-08-20T21:40:24.7978420Z remote: Compressing objects: 47% (398/846) 2024-08-20T21:40:24.7981256Z remote: Compressing objects: 48% (407/846) 2024-08-20T21:40:24.7983975Z remote: Compressing objects: 49% (415/846) 2024-08-20T21:40:24.7988158Z remote: Compressing objects: 50% (423/846) 2024-08-20T21:40:24.7990114Z remote: Compressing objects: 51% (432/846) 2024-08-20T21:40:24.7993495Z remote: Compressing objects: 52% (440/846) 2024-08-20T21:40:24.7995488Z remote: Compressing objects: 53% (449/846) 2024-08-20T21:40:24.7997665Z remote: Compressing objects: 54% (457/846) 2024-08-20T21:40:24.7999150Z remote: Compressing objects: 55% (466/846) 2024-08-20T21:40:24.8000802Z remote: Compressing objects: 56% (474/846) 2024-08-20T21:40:24.8001918Z remote: Compressing objects: 57% (483/846) 2024-08-20T21:40:24.8002612Z remote: Compressing objects: 58% (491/846) 2024-08-20T21:40:24.8003144Z remote: Compressing objects: 59% (500/846) 2024-08-20T21:40:24.8004283Z remote: Compressing objects: 60% (508/846) 2024-08-20T21:40:24.8004877Z remote: Compressing objects: 61% (517/846) 2024-08-20T21:40:24.8005565Z remote: Compressing objects: 62% (525/846) 2024-08-20T21:40:24.8006109Z remote: Compressing objects: 63% (533/846) 2024-08-20T21:40:24.8006640Z remote: Compressing objects: 64% (542/846) 2024-08-20T21:40:24.8011374Z remote: Compressing objects: 65% (550/846) 2024-08-20T21:40:24.8017724Z remote: Compressing objects: 66% (559/846) 2024-08-20T21:40:24.8022702Z remote: Compressing objects: 67% (567/846) 2024-08-20T21:40:24.8027534Z remote: Compressing objects: 68% (576/846) 2024-08-20T21:40:24.8032344Z remote: Compressing objects: 69% (584/846) 2024-08-20T21:40:24.8034739Z remote: Compressing objects: 70% (593/846) 2024-08-20T21:40:24.8039961Z remote: Compressing objects: 71% (601/846) 2024-08-20T21:40:24.8042826Z remote: Compressing objects: 72% (610/846) 2024-08-20T21:40:24.8045712Z remote: Compressing objects: 73% (618/846) 2024-08-20T21:40:24.8048957Z remote: Compressing objects: 74% (627/846) 2024-08-20T21:40:24.8051277Z remote: Compressing objects: 75% (635/846) 2024-08-20T21:40:24.8054632Z remote: Compressing objects: 76% (643/846) 2024-08-20T21:40:24.8057240Z remote: Compressing objects: 77% (652/846) 2024-08-20T21:40:24.8059848Z remote: Compressing objects: 78% (660/846) 2024-08-20T21:40:24.8062522Z remote: Compressing objects: 79% (669/846) 2024-08-20T21:40:24.8064859Z remote: Compressing objects: 80% (677/846) 2024-08-20T21:40:24.8065391Z remote: Compressing objects: 81% (686/846) 2024-08-20T21:40:24.8067244Z remote: Compressing objects: 82% (694/846) 2024-08-20T21:40:24.8069331Z remote: Compressing objects: 83% (703/846) 2024-08-20T21:40:24.8071314Z remote: Compressing objects: 84% (711/846) 2024-08-20T21:40:24.8072793Z remote: Compressing objects: 85% (720/846) 2024-08-20T21:40:24.8074730Z remote: Compressing objects: 86% (728/846) 2024-08-20T21:40:24.8076058Z remote: Compressing objects: 87% (737/846) 2024-08-20T21:40:24.8079884Z remote: Compressing objects: 88% (745/846) 2024-08-20T21:40:24.8081288Z remote: Compressing objects: 89% (753/846) 2024-08-20T21:40:24.8082037Z remote: Compressing objects: 90% (762/846) 2024-08-20T21:40:24.8083267Z remote: Compressing objects: 91% (770/846) 2024-08-20T21:40:24.8083803Z remote: Compressing objects: 92% (779/846) 2024-08-20T21:40:24.8084396Z remote: Compressing objects: 93% (787/846) 2024-08-20T21:40:24.8085177Z remote: Compressing objects: 94% (796/846) 2024-08-20T21:40:24.8086890Z remote: Compressing objects: 95% (804/846) 2024-08-20T21:40:24.8088299Z remote: Compressing objects: 96% (813/846) 2024-08-20T21:40:24.8089183Z remote: Compressing objects: 97% (821/846) 2024-08-20T21:40:24.8090382Z remote: Compressing objects: 98% (830/846) 2024-08-20T21:40:24.8091827Z remote: Compressing objects: 99% (838/846) 2024-08-20T21:40:24.8092361Z remote: Compressing objects: 100% (846/846) 2024-08-20T21:40:24.8092935Z remote: Compressing objects: 100% (846/846), done. 2024-08-20T21:40:44.8993813Z remote: Total 1010152 (delta 1120), reused 1324 (delta 850), pack-reused 1008453 (from 1) 2024-08-20T21:41:06.1411534Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T21:41:06.1440726Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:41:06.1444103Z ##[endgroup] 2024-08-20T21:41:06.1444685Z ##[group]Determining the checkout info 2024-08-20T21:41:06.1445580Z ##[endgroup] 2024-08-20T21:41:06.1446084Z ##[group]Checking out the ref 2024-08-20T21:41:06.1449233Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:41:07.4610710Z ##[endgroup] 2024-08-20T21:41:07.4615066Z ##[group]Setting up auth for fetching submodules 2024-08-20T21:41:07.4616205Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:41:07.4669139Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T21:41:07.4697779Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T21:41:07.4727099Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T21:41:07.4754476Z ##[endgroup] 2024-08-20T21:41:07.4754987Z ##[group]Fetching submodules 2024-08-20T21:41:07.4757695Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T21:41:07.5049549Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T21:41:07.5327103Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T21:41:07.5328600Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T21:41:07.5330063Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T21:41:07.5331598Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T21:41:07.5334563Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T21:41:07.5336315Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T21:41:07.5338923Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T21:41:07.5341591Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T21:41:07.5344215Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T21:41:07.5347555Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T21:41:07.5350346Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T21:41:07.5353310Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T21:41:07.5356413Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T21:41:07.5359810Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T21:41:07.5362931Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T21:41:07.5366546Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:41:07.5371401Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T21:41:07.5375267Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T21:41:07.5378736Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T21:41:07.5382685Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T21:41:07.5386475Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T21:41:07.5390505Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T21:41:07.5394410Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T21:41:07.5398561Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T21:41:07.5402650Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T21:41:07.5407309Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T21:41:07.5411526Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T21:41:07.5416134Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T21:41:07.5420688Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T21:41:07.5425450Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T21:41:07.5430050Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T21:41:07.5435051Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T21:41:07.5439726Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T21:41:07.5446552Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T21:41:07.5473742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T21:41:07.8573903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T21:41:08.0334791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T21:41:08.2014724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T21:41:08.4204500Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T21:41:10.1008832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T21:41:18.5556444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T21:41:18.9336437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T21:41:19.3383452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T21:41:19.9012388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T21:41:20.9647564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T21:41:22.8285291Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T21:41:27.6836926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T21:41:28.8728889Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T21:41:30.3033226Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T21:41:31.3021035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T21:41:31.6809198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T21:41:31.9976857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T21:41:32.8466060Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T21:41:33.1591491Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T21:41:33.4033916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T21:41:34.8554000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T21:41:35.4863239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T21:41:36.1690799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T21:41:42.4578764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T21:41:43.9932115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T21:41:46.9374161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T21:41:47.1638974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T21:41:54.6901262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T21:41:54.8655859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T21:41:55.0525162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T21:41:55.8103428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T21:41:56.0861896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T21:41:56.6543820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T21:41:57.0425746Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T21:41:57.0535062Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T21:41:57.0619074Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T21:41:57.0844085Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T21:41:57.1193477Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T21:41:58.0084337Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T21:41:58.0299603Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T21:41:58.0718905Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T21:41:58.1614771Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T21:41:58.1931311Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T21:41:58.6582528Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T21:41:58.8933684Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T21:41:58.9693667Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T21:41:58.9711476Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:41:58.9715253Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:41:58.9716842Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:41:58.9718428Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:41:58.9720190Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:41:58.9746213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T21:41:59.8466093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T21:42:00.4153956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T21:42:02.2003161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T21:42:03.0836622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T21:42:03.4278244Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T21:42:03.5168842Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T21:42:03.8887350Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T21:42:03.9471612Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T21:42:03.9594653Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T21:42:04.0678322Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T21:42:04.1051687Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T21:42:04.1412515Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T21:42:04.1643658Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T21:42:04.2056808Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:42:04.2174270Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T21:42:04.2190612Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:04.2214080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T21:42:15.0100372Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T21:42:15.0276405Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T21:42:15.1094691Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T21:42:15.1112450Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:15.1114269Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:15.1116062Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:15.1139847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T21:42:15.5892731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T21:42:16.5546450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T21:42:17.4778262Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T21:42:17.4796103Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:17.4798328Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:17.4800510Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:17.4802738Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:17.4804607Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:17.4806902Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:17.4809234Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:17.4811584Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:17.4839996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T21:42:18.2279210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T21:42:18.5787318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T21:42:19.5255817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T21:42:19.8355455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T21:42:20.3198984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T21:42:21.2195489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T21:42:26.1511841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T21:42:26.5639249Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T21:42:26.5815791Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T21:42:26.6158592Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T21:42:26.6285812Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T21:42:26.6299610Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:26.6326095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T21:42:26.8942158Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T21:42:26.9118128Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T21:42:26.9489511Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T21:42:27.0461041Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T21:42:27.0615339Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T21:42:27.1003158Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T21:42:27.1543396Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T21:42:27.1883496Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T21:42:27.2101457Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T21:42:27.3066683Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T21:42:27.6351155Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T21:42:27.6384343Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:27.6386040Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:27.6413996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T21:42:28.0414849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T21:42:28.8283645Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T21:42:28.8603030Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T21:42:28.9221344Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T21:42:28.9239731Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:28.9241525Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:28.9243215Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:28.9244894Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:28.9247354Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:28.9249736Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:28.9252549Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:28.9255123Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:28.9281731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T21:42:29.3111921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T21:42:30.2258450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T21:42:30.5316156Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T21:42:36.9021619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T21:42:37.1564747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T21:42:37.3375051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T21:42:37.6356451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T21:42:43.1084151Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T21:42:43.1475552Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T21:42:43.1621089Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T21:42:43.2538696Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T21:42:43.2658218Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T21:42:43.2803770Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T21:42:43.2948961Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T21:42:43.2964395Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:43.2966368Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:43.2990324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T21:42:44.8636616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T21:42:45.9752069Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T21:42:46.0191095Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:42:46.4528890Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T21:42:46.4644556Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T21:42:46.7116170Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T21:42:46.7135878Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:46.7137915Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:46.7166401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T21:42:47.1118246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T21:42:47.9922764Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T21:42:48.0577198Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T21:42:48.0662806Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T21:42:48.0780541Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T21:42:48.1121338Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T21:42:48.1397241Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T21:42:48.1782692Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T21:42:48.2028902Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T21:42:48.2044685Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:48.2046554Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:48.2048314Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:48.2049928Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:48.2075770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T21:42:49.0716971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T21:42:49.2775313Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T21:42:50.3249188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T21:42:51.1172608Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T21:42:51.1314628Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T21:42:51.1938216Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T21:42:51.2203340Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T21:42:51.2216313Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:51.2242264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T21:42:51.4549726Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T21:42:51.4591011Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T21:42:51.4907460Z Entering 'android/libs/fbjni' 2024-08-20T21:42:51.4947906Z Entering 'third_party/FP16' 2024-08-20T21:42:51.4990440Z Entering 'third_party/FXdiv' 2024-08-20T21:42:51.5032191Z Entering 'third_party/NNPACK' 2024-08-20T21:42:51.5074257Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:51.5115422Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:51.5174275Z Entering 'third_party/benchmark' 2024-08-20T21:42:51.5213463Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:51.5253868Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:51.5293274Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:51.5333088Z Entering 'third_party/cutlass' 2024-08-20T21:42:51.5378451Z Entering 'third_party/eigen' 2024-08-20T21:42:51.5421283Z Entering 'third_party/fbgemm' 2024-08-20T21:42:51.5460295Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:51.5498350Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:51.5536324Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:51.5581885Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:51.5619710Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:51.5658678Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:51.5699903Z Entering 'third_party/fmt' 2024-08-20T21:42:51.5743988Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:51.5782685Z Entering 'third_party/gloo' 2024-08-20T21:42:51.5823634Z Entering 'third_party/googletest' 2024-08-20T21:42:51.5861890Z Entering 'third_party/ideep' 2024-08-20T21:42:51.5908362Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:51.5960459Z Entering 'third_party/ittapi' 2024-08-20T21:42:51.5999758Z Entering 'third_party/kineto' 2024-08-20T21:42:51.6040950Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:51.6080849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:51.6120157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:51.6159707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:51.6198355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:51.6235424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:51.6276153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:51.6314388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:51.6350384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:51.6391024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:51.6435946Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:51.6475371Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:51.6515729Z Entering 'third_party/mimalloc' 2024-08-20T21:42:51.6553883Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:51.6593940Z Entering 'third_party/nlohmann' 2024-08-20T21:42:51.6634151Z Entering 'third_party/onnx' 2024-08-20T21:42:51.6688570Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:51.6727263Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:51.6769192Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:51.6808754Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:51.6845514Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:51.6887700Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:51.6925339Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:51.6964298Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:51.7003451Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:51.7041398Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:51.7083775Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:51.7123469Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:51.7164496Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:51.7224730Z Entering 'third_party/pocketfft' 2024-08-20T21:42:51.7264540Z Entering 'third_party/protobuf' 2024-08-20T21:42:51.7306953Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:51.7345469Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:51.7391206Z Entering 'third_party/psimd' 2024-08-20T21:42:51.7428785Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:51.7468126Z Entering 'third_party/pybind11' 2024-08-20T21:42:51.7507723Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:51.7547131Z Entering 'third_party/sleef' 2024-08-20T21:42:51.7587259Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:51.7628049Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:51.7667005Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:51.7703395Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:51.7748797Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:51.7787295Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:51.7849100Z ##[endgroup] 2024-08-20T21:42:51.7850972Z ##[group]Persisting credentials for submodules 2024-08-20T21:42:51.7852908Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T21:42:51.8134840Z Entering 'android/libs/fbjni' 2024-08-20T21:42:51.8188262Z Entering 'third_party/FP16' 2024-08-20T21:42:51.8241232Z Entering 'third_party/FXdiv' 2024-08-20T21:42:51.8292028Z Entering 'third_party/NNPACK' 2024-08-20T21:42:51.8341630Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:51.8394617Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:51.8466584Z Entering 'third_party/benchmark' 2024-08-20T21:42:51.8516022Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:51.8566125Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:51.8616443Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:51.8668663Z Entering 'third_party/cutlass' 2024-08-20T21:42:51.8726539Z Entering 'third_party/eigen' 2024-08-20T21:42:51.8778551Z Entering 'third_party/fbgemm' 2024-08-20T21:42:51.8829002Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:51.8883620Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:51.8932918Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:51.8998045Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:51.9047314Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:51.9097293Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:51.9149923Z Entering 'third_party/fmt' 2024-08-20T21:42:51.9202473Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:51.9251005Z Entering 'third_party/gloo' 2024-08-20T21:42:51.9300591Z Entering 'third_party/googletest' 2024-08-20T21:42:51.9349709Z Entering 'third_party/ideep' 2024-08-20T21:42:51.9400955Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:51.9455664Z Entering 'third_party/ittapi' 2024-08-20T21:42:51.9507432Z Entering 'third_party/kineto' 2024-08-20T21:42:51.9555093Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:51.9609052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:51.9659111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:51.9709521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:51.9757186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:51.9808927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:51.9861073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:51.9912192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:51.9964772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:52.0014297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:52.0066088Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:52.0114561Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.0166192Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.0215499Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.0267088Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.0317603Z Entering 'third_party/onnx' 2024-08-20T21:42:52.0381852Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.0432025Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.0493008Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.0542946Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.0595097Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.0647572Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.0699801Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.0750759Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.0801634Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.0849687Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.0898538Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.0949072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:52.1002735Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:52.1079180Z Entering 'third_party/pocketfft' 2024-08-20T21:42:52.1128860Z Entering 'third_party/protobuf' 2024-08-20T21:42:52.1181077Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:52.1231093Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:52.1284729Z Entering 'third_party/psimd' 2024-08-20T21:42:52.1333226Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:52.1383296Z Entering 'third_party/pybind11' 2024-08-20T21:42:52.1437058Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:52.1491682Z Entering 'third_party/sleef' 2024-08-20T21:42:52.1545053Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:52.1595544Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:52.1645635Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:52.1698785Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:52.1748098Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:52.1797881Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:52.1861984Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T21:42:52.2142733Z Entering 'android/libs/fbjni' 2024-08-20T21:42:52.2191689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T21:42:52.2208282Z Entering 'third_party/FP16' 2024-08-20T21:42:52.2255015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T21:42:52.2271461Z Entering 'third_party/FXdiv' 2024-08-20T21:42:52.2318152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T21:42:52.2332582Z Entering 'third_party/NNPACK' 2024-08-20T21:42:52.2379358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T21:42:52.2396365Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:52.2447853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T21:42:52.2461124Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:52.2510065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T21:42:52.2544818Z Entering 'third_party/benchmark' 2024-08-20T21:42:52.2591867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.2608528Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:52.2654757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T21:42:52.2669996Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:52.2716722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:42:52.2733268Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:52.2778580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T21:42:52.2795818Z Entering 'third_party/cutlass' 2024-08-20T21:42:52.2841879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:42:52.2867996Z Entering 'third_party/eigen' 2024-08-20T21:42:52.2913628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T21:42:52.2932811Z Entering 'third_party/fbgemm' 2024-08-20T21:42:52.2979347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T21:42:52.2995981Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:52.3040981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T21:42:52.3054767Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:52.3101506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:42:52.3117961Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:52.3164586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:42:52.3187489Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:52.3234717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.3248527Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:52.3293980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T21:42:52.3312166Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:52.3358918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T21:42:52.3374998Z Entering 'third_party/fmt' 2024-08-20T21:42:52.3420331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:42:52.3437077Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:52.3488643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T21:42:52.3502022Z Entering 'third_party/gloo' 2024-08-20T21:42:52.3548201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T21:42:52.3564441Z Entering 'third_party/googletest' 2024-08-20T21:42:52.3610741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.3628260Z Entering 'third_party/ideep' 2024-08-20T21:42:52.3675488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T21:42:52.3690580Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:52.3734794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T21:42:52.3758584Z Entering 'third_party/ittapi' 2024-08-20T21:42:52.3803834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T21:42:52.3817568Z Entering 'third_party/kineto' 2024-08-20T21:42:52.3866067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T21:42:52.3881839Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:52.3927978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T21:42:52.3941229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:52.3989870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T21:42:52.4006456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:52.4054282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T21:42:52.4070345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:52.4116613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:42:52.4132341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:52.4188741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T21:42:52.4202549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:52.4248440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T21:42:52.4266559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:52.4312719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T21:42:52.4328107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:52.4374226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.4390290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:52.4439148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T21:42:52.4453542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:52.4502216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T21:42:52.4519304Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:52.4565896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T21:42:52.4579883Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.4628162Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.4646104Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.4692230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T21:42:52.4708647Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.4752723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T21:42:52.4769359Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.4813833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T21:42:52.4831739Z Entering 'third_party/onnx' 2024-08-20T21:42:52.4880457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T21:42:52.4910984Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.4956653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.4972397Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.5019605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.5039739Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.5087506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T21:42:52.5102239Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.5149518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.5165280Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.5212020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.5228189Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.5274769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T21:42:52.5290250Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.5334581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T21:42:52.5350225Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.5399144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T21:42:52.5412983Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.5457865Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T21:42:52.5473802Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.5526013Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T21:42:52.5538514Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.5587112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T21:42:52.5604372Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:52.5650127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T21:42:52.5668805Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:52.5715634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T21:42:52.5749200Z Entering 'third_party/pocketfft' 2024-08-20T21:42:52.5796902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T21:42:52.5812900Z Entering 'third_party/protobuf' 2024-08-20T21:42:52.5856825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T21:42:52.5876507Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:52.5921921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:42:52.5935640Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:52.5982750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.6002498Z Entering 'third_party/psimd' 2024-08-20T21:42:52.6047347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T21:42:52.6060470Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:52.6108478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T21:42:52.6123380Z Entering 'third_party/pybind11' 2024-08-20T21:42:52.6172043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.6188863Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:52.6235237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T21:42:52.6249789Z Entering 'third_party/sleef' 2024-08-20T21:42:52.6294338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T21:42:52.6311212Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:52.6357464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T21:42:52.6373070Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:52.6417249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:42:52.6433325Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:52.6480890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T21:42:52.6493611Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:52.6541837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T21:42:52.6554569Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:52.6602805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:42:52.6615092Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:52.6662304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T21:42:52.7501127Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T21:42:52.7788152Z Entering 'android/libs/fbjni' 2024-08-20T21:42:52.7831312Z Entering 'third_party/FP16' 2024-08-20T21:42:52.7871250Z Entering 'third_party/FXdiv' 2024-08-20T21:42:52.7909897Z Entering 'third_party/NNPACK' 2024-08-20T21:42:52.7947413Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:52.7987536Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:52.8041341Z Entering 'third_party/benchmark' 2024-08-20T21:42:52.8083738Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:52.8123374Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:52.8163973Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:52.8203339Z Entering 'third_party/cutlass' 2024-08-20T21:42:52.8248691Z Entering 'third_party/eigen' 2024-08-20T21:42:52.8292777Z Entering 'third_party/fbgemm' 2024-08-20T21:42:52.8332698Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:52.8369748Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:52.8408322Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:52.8451991Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:52.8497141Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:52.8535758Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:52.8577430Z Entering 'third_party/fmt' 2024-08-20T21:42:52.8615817Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:52.8655938Z Entering 'third_party/gloo' 2024-08-20T21:42:52.8695169Z Entering 'third_party/googletest' 2024-08-20T21:42:52.8733794Z Entering 'third_party/ideep' 2024-08-20T21:42:52.8773350Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:52.8816692Z Entering 'third_party/ittapi' 2024-08-20T21:42:52.8855708Z Entering 'third_party/kineto' 2024-08-20T21:42:52.8894600Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:52.8933494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:52.8973124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:52.9012037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:52.9053435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:52.9091984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:52.9133002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:52.9171251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:52.9209457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:52.9247736Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:52.9288293Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:52.9327104Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:52.9367378Z Entering 'third_party/mimalloc' 2024-08-20T21:42:52.9406696Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:52.9446052Z Entering 'third_party/nlohmann' 2024-08-20T21:42:52.9485987Z Entering 'third_party/onnx' 2024-08-20T21:42:52.9538162Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:52.9576880Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:52.9621057Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:52.9663956Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:52.9701419Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:52.9738062Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:52.9776896Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:52.9816096Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:52.9853426Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:52.9898101Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:52.9935358Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:52.9975920Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:53.0015042Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:53.0081513Z Entering 'third_party/pocketfft' 2024-08-20T21:42:53.0120335Z Entering 'third_party/protobuf' 2024-08-20T21:42:53.0163636Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:53.0205259Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:53.0246755Z Entering 'third_party/psimd' 2024-08-20T21:42:53.0286005Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:53.0326062Z Entering 'third_party/pybind11' 2024-08-20T21:42:53.0365307Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:53.0408678Z Entering 'third_party/sleef' 2024-08-20T21:42:53.0449907Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:53.0490146Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:53.0527628Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:53.0565986Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:53.0607560Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:53.0645161Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:53.0707216Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T21:42:53.0999994Z Entering 'android/libs/fbjni' 2024-08-20T21:42:53.1038876Z Entering 'third_party/FP16' 2024-08-20T21:42:53.1077712Z Entering 'third_party/FXdiv' 2024-08-20T21:42:53.1115856Z Entering 'third_party/NNPACK' 2024-08-20T21:42:53.1153665Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:42:53.1193601Z Entering 'third_party/XNNPACK' 2024-08-20T21:42:53.1247443Z Entering 'third_party/benchmark' 2024-08-20T21:42:53.1286513Z Entering 'third_party/cpp-httplib' 2024-08-20T21:42:53.1327081Z Entering 'third_party/cpuinfo' 2024-08-20T21:42:53.1366331Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:42:53.1404549Z Entering 'third_party/cutlass' 2024-08-20T21:42:53.1449637Z Entering 'third_party/eigen' 2024-08-20T21:42:53.1493371Z Entering 'third_party/fbgemm' 2024-08-20T21:42:53.1532616Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:42:53.1570061Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:42:53.1608819Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:42:53.1658956Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:42:53.1696201Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:42:53.1737307Z Entering 'third_party/flatbuffers' 2024-08-20T21:42:53.1791702Z Entering 'third_party/fmt' 2024-08-20T21:42:53.1830089Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:42:53.1870002Z Entering 'third_party/gloo' 2024-08-20T21:42:53.1909957Z Entering 'third_party/googletest' 2024-08-20T21:42:53.1948049Z Entering 'third_party/ideep' 2024-08-20T21:42:53.1987973Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:42:53.2037114Z Entering 'third_party/ittapi' 2024-08-20T21:42:53.2076671Z Entering 'third_party/kineto' 2024-08-20T21:42:53.2115595Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:42:53.2156351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:42:53.2195214Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:42:53.2233456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:42:53.2272587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:42:53.2309134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:42:53.2348574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:42:53.2388441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:42:53.2427812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:42:53.2466896Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:42:53.2506630Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:42:53.2545148Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:42:53.2583255Z Entering 'third_party/mimalloc' 2024-08-20T21:42:53.2622788Z Entering 'third_party/nccl/nccl' 2024-08-20T21:42:53.2661496Z Entering 'third_party/nlohmann' 2024-08-20T21:42:53.2711716Z Entering 'third_party/onnx' 2024-08-20T21:42:53.2765261Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:42:53.2803596Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:42:53.2846511Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:42:53.2887069Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:42:53.2925995Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:42:53.2965021Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:42:53.3006138Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:42:53.3047630Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:42:53.3086054Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:42:53.3124260Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:42:53.3170944Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:42:53.3217939Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:42:53.3261114Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:42:53.3320574Z Entering 'third_party/pocketfft' 2024-08-20T21:42:53.3360798Z Entering 'third_party/protobuf' 2024-08-20T21:42:53.3402483Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:42:53.3441215Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:42:53.3482640Z Entering 'third_party/psimd' 2024-08-20T21:42:53.3521664Z Entering 'third_party/pthreadpool' 2024-08-20T21:42:53.3563104Z Entering 'third_party/pybind11' 2024-08-20T21:42:53.3601865Z Entering 'third_party/python-peachpy' 2024-08-20T21:42:53.3642088Z Entering 'third_party/sleef' 2024-08-20T21:42:53.3682235Z Entering 'third_party/tensorpipe' 2024-08-20T21:42:53.3721816Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:42:53.3757405Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:42:53.3795428Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:42:53.3832549Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:42:53.3869779Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:42:53.3923265Z ##[endgroup] 2024-08-20T21:42:53.3959028Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T21:42:53.3986641Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T21:42:53.4169084Z Prepare all required actions 2024-08-20T21:42:53.4169746Z Getting action download info 2024-08-20T21:42:53.5690963Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T21:42:53.5691462Z env: 2024-08-20T21:42:53.5691905Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.5692344Z ##[endgroup] 2024-08-20T21:42:53.5752280Z ##[group]Run set -euo pipefail 2024-08-20T21:42:53.5752844Z set -euo pipefail 2024-08-20T21:42:53.5753220Z function get_ec2_metadata() { 2024-08-20T21:42:53.5753749Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T21:42:53.5754664Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T21:42:53.5755457Z  category=$1 2024-08-20T21:42:53.5755981Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T21:42:53.5756605Z  runner_name_str=i-0315eba2eab835699 2024-08-20T21:42:53.5757115Z  if [[ -f /.inarc ]]; then 2024-08-20T21:42:53.5757606Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T21:42:53.5758270Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T21:42:53.5758919Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T21:42:53.5759492Z  else 2024-08-20T21:42:53.5759956Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T21:42:53.5760512Z  fi 2024-08-20T21:42:53.5760776Z } 2024-08-20T21:42:53.5761104Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T21:42:53.5761646Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T21:42:53.5762246Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T21:42:53.5762776Z echo "system info $(uname -a)" 2024-08-20T21:42:53.5770520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.5771015Z env: 2024-08-20T21:42:53.5771405Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.5771918Z ##[endgroup] 2024-08-20T21:42:53.5865235Z ami-id: ami-06c68f701d8090592 2024-08-20T21:42:53.5925413Z instance-id: i-0315eba2eab835699 2024-08-20T21:42:53.5985140Z instance-type: c5.12xlarge 2024-08-20T21:42:53.5995101Z system info Linux ip-10-0-17-252.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T21:42:53.6016857Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:42:53.6017789Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:42:53.6023634Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.6024217Z env: 2024-08-20T21:42:53.6024445Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.6024762Z ##[endgroup] 2024-08-20T21:42:53.6086440Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T21:42:53.6087005Z if systemctl is-active --quiet docker; then 2024-08-20T21:42:53.6087505Z  echo "Docker daemon is running..."; 2024-08-20T21:42:53.6087907Z else 2024-08-20T21:42:53.6088371Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T21:42:53.6088911Z fi 2024-08-20T21:42:53.6094532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:53.6095034Z env: 2024-08-20T21:42:53.6095294Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.6095629Z ##[endgroup] 2024-08-20T21:42:53.6167593Z Docker daemon is running... 2024-08-20T21:42:53.6210438Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:42:53.6210989Z with: 2024-08-20T21:42:53.6211347Z shell: bash 2024-08-20T21:42:53.6211787Z timeout_minutes: 5 2024-08-20T21:42:53.6212111Z max_attempts: 3 2024-08-20T21:42:53.6212410Z retry_wait_seconds: 30 2024-08-20T21:42:53.6215717Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T21:42:53.6219038Z polling_interval_seconds: 1 2024-08-20T21:42:53.6219389Z warning_on_retry: true 2024-08-20T21:42:53.6219727Z continue_on_error: false 2024-08-20T21:42:53.6220058Z env: 2024-08-20T21:42:53.6220314Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:53.6220638Z AWS_RETRY_MODE: standard 2024-08-20T21:42:53.6220974Z AWS_MAX_ATTEMPTS: 5 2024-08-20T21:42:53.6221303Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T21:42:53.6221666Z ##[endgroup] 2024-08-20T21:42:54.6884958Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:54.6885839Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:54.6886741Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:54.6887329Z 2024-08-20T21:42:54.6887435Z Login Succeeded 2024-08-20T21:42:55.6751320Z Command completed after 1 attempt(s). 2024-08-20T21:42:55.6808089Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.6808811Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.6809461Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:42:55.6816092Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.6816579Z env: 2024-08-20T21:42:55.6816855Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.6817202Z ##[endgroup] 2024-08-20T21:42:55.6904952Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:42:55.6905721Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:42:55.6906307Z # shellcheck disable=SC2046 2024-08-20T21:42:55.6906755Z docker stop $(docker ps -q) || true 2024-08-20T21:42:55.6907203Z # Prune all of the docker images 2024-08-20T21:42:55.6907650Z docker system prune -af 2024-08-20T21:42:55.6913735Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.6914224Z env: 2024-08-20T21:42:55.6914475Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.6914812Z ##[endgroup] 2024-08-20T21:42:55.7201199Z "docker stop" requires at least 1 argument. 2024-08-20T21:42:55.7201917Z See 'docker stop --help'. 2024-08-20T21:42:55.7202175Z 2024-08-20T21:42:55.7202393Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T21:42:55.7202764Z 2024-08-20T21:42:55.7202922Z Stop one or more running containers 2024-08-20T21:42:55.7514239Z Deleted Images: 2024-08-20T21:42:55.7515010Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine:latest 2024-08-20T21:42:55.7516351Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine@sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748 2024-08-20T21:42:55.7517563Z deleted: sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec 2024-08-20T21:42:55.7518471Z deleted: sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 2024-08-20T21:42:55.7519024Z 2024-08-20T21:42:55.7531971Z Total reclaimed space: 5.613MB 2024-08-20T21:42:55.7653589Z ##[group]Run set +e 2024-08-20T21:42:55.7653934Z set +e 2024-08-20T21:42:55.7654208Z set -x 2024-08-20T21:42:55.7654492Z  2024-08-20T21:42:55.7654804Z PT_DOMAIN=download.pytorch.org 2024-08-20T21:42:55.7655607Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T21:42:55.7656854Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T21:42:55.7657606Z # one is returned at random 2024-08-20T21:42:55.7658145Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T21:42:55.7658656Z  2024-08-20T21:42:55.7658957Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:42:55.7659829Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T21:42:55.7660586Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T21:42:55.7661140Z  2024-08-20T21:42:55.7661445Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:42:55.7661979Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T21:42:55.7662482Z  exit 1 2024-08-20T21:42:55.7662777Z  fi 2024-08-20T21:42:55.7663030Z fi 2024-08-20T21:42:55.7663386Z  2024-08-20T21:42:55.7663706Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T21:42:55.7664176Z  # Clean up any old records first 2024-08-20T21:42:55.7664628Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T21:42:55.7665033Z fi 2024-08-20T21:42:55.7665274Z  2024-08-20T21:42:55.7665658Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T21:42:55.7666150Z cat /etc/hosts 2024-08-20T21:42:55.7671651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:55.7672109Z env: 2024-08-20T21:42:55.7672359Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:55.7672667Z ##[endgroup] 2024-08-20T21:42:55.7696544Z + PT_DOMAIN=download.pytorch.org 2024-08-20T21:42:55.7701823Z ++ dig -4 +short download.pytorch.org 2024-08-20T21:42:55.7702492Z ++ tail -n1 2024-08-20T21:42:55.7972274Z + RESOLVED_IP=18.165.98.58 2024-08-20T21:42:55.7972878Z + '[' -z 18.165.98.58 ']' 2024-08-20T21:42:55.7973320Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T21:42:55.7983952Z 18.160.10.28 download.pytorch.org 2024-08-20T21:42:55.7985533Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T21:42:56.0876895Z + echo '18.165.98.58 download.pytorch.org' 2024-08-20T21:42:56.0877435Z + sudo tee -a /etc/hosts 2024-08-20T21:42:56.1260517Z 18.165.98.58 download.pytorch.org 2024-08-20T21:42:56.1273702Z + cat /etc/hosts 2024-08-20T21:42:56.1282483Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T21:42:56.1294166Z ::1 localhost6 localhost6.localdomain6 2024-08-20T21:42:56.1294801Z 18.165.98.58 download.pytorch.org 2024-08-20T21:42:56.1451355Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T21:42:56.1451956Z with: 2024-08-20T21:42:56.1452995Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1454175Z docker-build-dir: .ci/docker 2024-08-20T21:42:56.1454536Z working-directory: . 2024-08-20T21:42:56.1454993Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.1455511Z force-push: false 2024-08-20T21:42:56.1455792Z env: 2024-08-20T21:42:56.1456042Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.1456362Z ##[endgroup] 2024-08-20T21:42:56.1473408Z ##[group]Run set -ex 2024-08-20T21:42:56.1473749Z set -ex 2024-08-20T21:42:56.1473994Z  2024-08-20T21:42:56.1474508Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T21:42:56.1475476Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T21:42:56.1476219Z # job could then download the pre-built image as usual 2024-08-20T21:42:56.1476917Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T21:42:56.1477542Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1478130Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1478648Z  2024-08-20T21:42:56.1479105Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T21:42:56.1479691Z  exit 0 2024-08-20T21:42:56.1479957Z else 2024-08-20T21:42:56.1480385Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1480798Z fi 2024-08-20T21:42:56.1481039Z  2024-08-20T21:42:56.1481460Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T21:42:56.1482257Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T21:42:56.1482984Z  # use it as it is, but first let's extract the tag 2024-08-20T21:42:56.1483636Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T21:42:56.1484313Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1484963Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1485465Z else 2024-08-20T21:42:56.1485857Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T21:42:56.1486467Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1487308Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1488008Z fi 2024-08-20T21:42:56.1494932Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.1495424Z env: 2024-08-20T21:42:56.1495679Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.1496015Z REPO_NAME: pytorch 2024-08-20T21:42:56.1497078Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1498240Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:42:56.1498727Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.1499232Z ##[endgroup] 2024-08-20T21:42:56.1522314Z + [[ ! -d .ci/docker ]] 2024-08-20T21:42:56.1522740Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T21:42:56.1523180Z + echo skip=false 2024-08-20T21:42:56.1524974Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T21:42:56.1529607Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1530778Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T21:42:56.1563314Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1564108Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1565665Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1589723Z ##[group]Run set +e 2024-08-20T21:42:56.1590052Z set +e 2024-08-20T21:42:56.1590316Z set -x 2024-08-20T21:42:56.1590566Z  2024-08-20T21:42:56.1590804Z login() { 2024-08-20T21:42:56.1591409Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:42:56.1592076Z } 2024-08-20T21:42:56.1592305Z  2024-08-20T21:42:56.1592543Z retry () { 2024-08-20T21:42:56.1592887Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:42:56.1593292Z } 2024-08-20T21:42:56.1593518Z  2024-08-20T21:42:56.1593791Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:42:56.1594160Z  2024-08-20T21:42:56.1594574Z # Check if image already exists, if it does then skip building it 2024-08-20T21:42:56.1595225Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T21:42:56.1595680Z  exit 0 2024-08-20T21:42:56.1595939Z fi 2024-08-20T21:42:56.1596163Z  2024-08-20T21:42:56.1596618Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T21:42:56.1597513Z # be empty. The default action would be to continue rebuild the image 2024-08-20T21:42:56.1598197Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T21:42:56.1598807Z  # if we're on the base branch then use the parent commit 2024-08-20T21:42:56.1599517Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T21:42:56.1599927Z else 2024-08-20T21:42:56.1600368Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T21:42:56.1601027Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T21:42:56.1601512Z fi 2024-08-20T21:42:56.1601765Z  2024-08-20T21:42:56.1602157Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T21:42:56.1602598Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1603009Z  2024-08-20T21:42:56.1603613Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T21:42:56.1604335Z  exit 0 2024-08-20T21:42:56.1604583Z fi 2024-08-20T21:42:56.1604821Z  2024-08-20T21:42:56.1605212Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T21:42:56.1606122Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T21:42:56.1606865Z  exit 1 2024-08-20T21:42:56.1607125Z fi 2024-08-20T21:42:56.1607527Z  2024-08-20T21:42:56.1607988Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T21:42:56.1608915Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T21:42:56.1609770Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T21:42:56.1610721Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T21:42:56.1612103Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T21:42:56.1612765Z fi 2024-08-20T21:42:56.1613119Z  2024-08-20T21:42:56.1613458Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:42:56.1618920Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.1619410Z env: 2024-08-20T21:42:56.1619656Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.1620003Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:42:56.1620456Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:42:56.1621695Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1622901Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.1623501Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.1624127Z ##[endgroup] 2024-08-20T21:42:56.1646536Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.1647204Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.1648716Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:42:56.1649822Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.6694399Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:56.6695227Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:56.6696089Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:56.6696615Z 2024-08-20T21:42:56.6696741Z Login Succeeded 2024-08-20T21:42:56.6709416Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.8662152Z { 2024-08-20T21:42:56.8662531Z "schemaVersion": 2, 2024-08-20T21:42:56.8663123Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T21:42:56.8663719Z "config": { 2024-08-20T21:42:56.8664177Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T21:42:56.8664746Z "size": 43916, 2024-08-20T21:42:56.8665293Z "digest": "sha256:822356886a68936cce34917f5ffc60d864918fab7594bb08c26bf1816dd6110d" 2024-08-20T21:42:56.8665907Z }, 2024-08-20T21:42:56.8666152Z "layers": [ 2024-08-20T21:42:56.8666417Z { 2024-08-20T21:42:56.8666858Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8667441Z "size": 30440714, 2024-08-20T21:42:56.8667990Z "digest": "sha256:762bedf4b1b784c3de6c5022c5307d63123d3b7cdd59211317e37e9d477deaa0" 2024-08-20T21:42:56.8668629Z }, 2024-08-20T21:42:56.8668867Z { 2024-08-20T21:42:56.8669308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8669873Z "size": 1822, 2024-08-20T21:42:56.8670427Z "digest": "sha256:347beb78d39bc9079a3c20e891908ad5695cdea4a312b0e1de586110bb9ee72f" 2024-08-20T21:42:56.8671061Z }, 2024-08-20T21:42:56.8671307Z { 2024-08-20T21:42:56.8671744Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8672314Z "size": 313164873, 2024-08-20T21:42:56.8673124Z "digest": "sha256:71b614615bcb09bc43ad35a0238b1b82945122c70d95efa9728a6a044a7ddb73" 2024-08-20T21:42:56.8673759Z }, 2024-08-20T21:42:56.8673980Z { 2024-08-20T21:42:56.8674419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8674993Z "size": 803, 2024-08-20T21:42:56.8675522Z "digest": "sha256:f96c8558662d14e2cc064f94a0166dbf827cf4b783c8d3edb4dbc19dcfff8ca3" 2024-08-20T21:42:56.8676160Z }, 2024-08-20T21:42:56.8676400Z { 2024-08-20T21:42:56.8676835Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8677377Z "size": 106, 2024-08-20T21:42:56.8677945Z "digest": "sha256:c81ff9d20d36fbd48a567918c08f020d23bd83515aec7d23fda14cd56786be81" 2024-08-20T21:42:56.8678582Z }, 2024-08-20T21:42:56.8678820Z { 2024-08-20T21:42:56.8681390Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8682082Z "size": 544, 2024-08-20T21:42:56.8682644Z "digest": "sha256:151e9a9d289467dccd8eefb461af7188b39953d6ac10b678133551bd20586ce6" 2024-08-20T21:42:56.8683342Z }, 2024-08-20T21:42:56.8683583Z { 2024-08-20T21:42:56.8684031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8684605Z "size": 1214, 2024-08-20T21:42:56.8685164Z "digest": "sha256:8538dcd89f0cac92de23cde0e22b0e7cfde631fbef3062f6387bbc78c4b8c051" 2024-08-20T21:42:56.8685826Z }, 2024-08-20T21:42:56.8686073Z { 2024-08-20T21:42:56.8686521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8687076Z "size": 483, 2024-08-20T21:42:56.8687633Z "digest": "sha256:cd13007f6f0922ba40ddf0f45d8693c116a5f27e894bd04a34e40eb57bd5c640" 2024-08-20T21:42:56.8688286Z }, 2024-08-20T21:42:56.8688533Z { 2024-08-20T21:42:56.8688958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8689534Z "size": 110321925, 2024-08-20T21:42:56.8690118Z "digest": "sha256:74fd96b6ebd997dccc9767b9222e3138502d240bca371c2dab11ed193140d9cb" 2024-08-20T21:42:56.8690739Z }, 2024-08-20T21:42:56.8690987Z { 2024-08-20T21:42:56.8691577Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8692160Z "size": 3840, 2024-08-20T21:42:56.8692755Z "digest": "sha256:ce4e1800db98e945fe1b38aac9ac5afcc6edcf1c9cc321499bcb108d04fe1442" 2024-08-20T21:42:56.8693420Z }, 2024-08-20T21:42:56.8693667Z { 2024-08-20T21:42:56.8694114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8694662Z "size": 1910, 2024-08-20T21:42:56.8695197Z "digest": "sha256:402b067f30bf9e459448062a2a55e42169b09275d1fa427073a332adc6929112" 2024-08-20T21:42:56.8695947Z }, 2024-08-20T21:42:56.8696171Z { 2024-08-20T21:42:56.8696612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8697176Z "size": 701, 2024-08-20T21:42:56.8697753Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.8698389Z }, 2024-08-20T21:42:56.8698630Z { 2024-08-20T21:42:56.8699065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8699633Z "size": 2663407704, 2024-08-20T21:42:56.8700207Z "digest": "sha256:c6e13ff8aa9a65d1d347f8a336df5ca7febab9d34a3027c2b3c138c1a711baf4" 2024-08-20T21:42:56.8700851Z }, 2024-08-20T21:42:56.8701091Z { 2024-08-20T21:42:56.8701527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8702074Z "size": 32, 2024-08-20T21:42:56.8702625Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8703267Z }, 2024-08-20T21:42:56.8703488Z { 2024-08-20T21:42:56.8703921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8704479Z "size": 379, 2024-08-20T21:42:56.8705054Z "digest": "sha256:e238df05ddcc0ec1e1ce769b25f7103f76af6aa35e4aa4ccd1adcdb7621158f6" 2024-08-20T21:42:56.8705686Z }, 2024-08-20T21:42:56.8705925Z { 2024-08-20T21:42:56.8706358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8706921Z "size": 222887, 2024-08-20T21:42:56.8707452Z "digest": "sha256:a88675bdc72381c487323d232044cca10dd1f77e62eaf00a650bd28d27d6f9bf" 2024-08-20T21:42:56.8708073Z }, 2024-08-20T21:42:56.8708312Z { 2024-08-20T21:42:56.8708732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8709291Z "size": 231, 2024-08-20T21:42:56.8709835Z "digest": "sha256:f4cb80c411d1b5fb90688374327599a7de6ea89c800473afb1b9766003d5a39a" 2024-08-20T21:42:56.8710460Z }, 2024-08-20T21:42:56.8710688Z { 2024-08-20T21:42:56.8711126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8711692Z "size": 2933275, 2024-08-20T21:42:56.8712358Z "digest": "sha256:c9589ebd9ae736462b4dc9ea4763b2892eae668c19ca321e7c9ef2762b86a528" 2024-08-20T21:42:56.8712993Z }, 2024-08-20T21:42:56.8713239Z { 2024-08-20T21:42:56.8713681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8714251Z "size": 1805, 2024-08-20T21:42:56.8714794Z "digest": "sha256:dd51c681db1409845c172ac4fc813dfc9db27bc797fe919a47724410acd481c4" 2024-08-20T21:42:56.8715431Z }, 2024-08-20T21:42:56.8715672Z { 2024-08-20T21:42:56.8716091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8716660Z "size": 105, 2024-08-20T21:42:56.8717207Z "digest": "sha256:fc983e33a069ae2891a2161f9d24540f9e2728dddc9fd0e63c5b82454ab51f61" 2024-08-20T21:42:56.8717842Z }, 2024-08-20T21:42:56.8718072Z { 2024-08-20T21:42:56.8718506Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8719078Z "size": 564, 2024-08-20T21:42:56.8719624Z "digest": "sha256:bc705d688c776732ddd22c3540b0e135d7dc23e4d46b64918be11b2fa1a265c5" 2024-08-20T21:42:56.8720236Z }, 2024-08-20T21:42:56.8720480Z { 2024-08-20T21:42:56.8720915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8721461Z "size": 32, 2024-08-20T21:42:56.8722010Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8722647Z }, 2024-08-20T21:42:56.8722885Z { 2024-08-20T21:42:56.8723302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8723867Z "size": 104, 2024-08-20T21:42:56.8724404Z "digest": "sha256:99f6c1ac758215178233da5104ddb298caf3ae441129209a625e9e649463b980" 2024-08-20T21:42:56.8725024Z }, 2024-08-20T21:42:56.8725278Z { 2024-08-20T21:42:56.8725714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8726356Z "size": 504, 2024-08-20T21:42:56.8726922Z "digest": "sha256:c409d27a5a9560bf329dec2ed58b701a43f1ffa8babc7ca13c61321cf22ef007" 2024-08-20T21:42:56.8727572Z }, 2024-08-20T21:42:56.8727819Z { 2024-08-20T21:42:56.8728251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8728827Z "size": 101109909, 2024-08-20T21:42:56.8729418Z "digest": "sha256:a3a398ca1120cf9319dbbb81b12ddc41f7bf3b5bd3f3612c17c2bf7b5194e505" 2024-08-20T21:42:56.8730056Z }, 2024-08-20T21:42:56.8730304Z { 2024-08-20T21:42:56.8730750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8731424Z "size": 109, 2024-08-20T21:42:56.8731957Z "digest": "sha256:314875f5fae452a321bd50b8527ebb8cf7db0a53ff30293472014390479fcf64" 2024-08-20T21:42:56.8732597Z }, 2024-08-20T21:42:56.8732843Z { 2024-08-20T21:42:56.8733286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8733847Z "size": 490, 2024-08-20T21:42:56.8734413Z "digest": "sha256:8a5e90dafefa97ef114ac2d4c2f8e000536425318010f40d15bfdc6f8d7e2a79" 2024-08-20T21:42:56.8735056Z }, 2024-08-20T21:42:56.8735289Z { 2024-08-20T21:42:56.8735738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8736307Z "size": 333, 2024-08-20T21:42:56.8736866Z "digest": "sha256:e3b946ab3378ddc5e96c6355bd0aaa135f29b1ea34b417652442c9df42b6eb49" 2024-08-20T21:42:56.8737488Z }, 2024-08-20T21:42:56.8737849Z { 2024-08-20T21:42:56.8738260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8738783Z "size": 103, 2024-08-20T21:42:56.8739263Z "digest": "sha256:14a85a990f7189af056863e625917e6a5583ce8a7b1d319783011803ea1a9da4" 2024-08-20T21:42:56.8739847Z }, 2024-08-20T21:42:56.8740074Z { 2024-08-20T21:42:56.8740483Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8740997Z "size": 1473, 2024-08-20T21:42:56.8741501Z "digest": "sha256:16d809b167d49638350b86af396a05eca356ba903ca4953a083d29f28154c16a" 2024-08-20T21:42:56.8742073Z }, 2024-08-20T21:42:56.8742281Z { 2024-08-20T21:42:56.8742689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8743307Z "size": 611525039, 2024-08-20T21:42:56.8743848Z "digest": "sha256:9b1a95eedb8338e08a9ba00b933ef64420d41664b1a4c2a40f1703c79a3d2ba1" 2024-08-20T21:42:56.8744422Z }, 2024-08-20T21:42:56.8744653Z { 2024-08-20T21:42:56.8745480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8746208Z "size": 164, 2024-08-20T21:42:56.8746834Z "digest": "sha256:b35ca05427236370ed98f905cf9b8a0e4599173ef1843504fc2547e9525cd483" 2024-08-20T21:42:56.8747461Z }, 2024-08-20T21:42:56.8747708Z { 2024-08-20T21:42:56.8748130Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8748694Z "size": 2210, 2024-08-20T21:42:56.8749254Z "digest": "sha256:bad10ac3a6279679db06f845b321a9fdaef80be969acbfd257f9e807762953a8" 2024-08-20T21:42:56.8749903Z }, 2024-08-20T21:42:56.8750128Z { 2024-08-20T21:42:56.8750563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8751132Z "size": 152, 2024-08-20T21:42:56.8751689Z "digest": "sha256:943da2164c7a7bf690bcbc751afe5ea646925ddf0ab96fd2ce48b0a52bd66e9d" 2024-08-20T21:42:56.8752314Z }, 2024-08-20T21:42:56.8752552Z { 2024-08-20T21:42:56.8752985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8753548Z "size": 648, 2024-08-20T21:42:56.8754077Z "digest": "sha256:2d276441bd8ae528772f4fd876af429ab3518bebf61090795ad3b6aced9d60e4" 2024-08-20T21:42:56.8754705Z }, 2024-08-20T21:42:56.8754944Z { 2024-08-20T21:42:56.8755361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8755921Z "size": 32, 2024-08-20T21:42:56.8756469Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8757270Z }, 2024-08-20T21:42:56.8757489Z { 2024-08-20T21:42:56.8757921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8758595Z "size": 167, 2024-08-20T21:42:56.8759144Z "digest": "sha256:a3ef8d7d02be75588561825a22ecbdc3129ad8a543f4d1415fee8284d91327bb" 2024-08-20T21:42:56.8759747Z }, 2024-08-20T21:42:56.8759983Z { 2024-08-20T21:42:56.8760406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8760939Z "size": 109, 2024-08-20T21:42:56.8761479Z "digest": "sha256:6aaed7a7874900b7539103d4ef762ab0f9a2d760c898f2ebcf135b73858aeed5" 2024-08-20T21:42:56.8762087Z }, 2024-08-20T21:42:56.8762320Z { 2024-08-20T21:42:56.8762725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8763269Z "size": 102, 2024-08-20T21:42:56.8763796Z "digest": "sha256:944e780839169e63df50b5cc05cc9fb99e8ed0b9692d56cee9cd06137bdb08e3" 2024-08-20T21:42:56.8764404Z }, 2024-08-20T21:42:56.8764625Z { 2024-08-20T21:42:56.8765051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8765595Z "size": 422, 2024-08-20T21:42:56.8766120Z "digest": "sha256:9602de36240c11e5eb9a1926c6443a2ff43948a958a15162e7aa827e1f83ad3e" 2024-08-20T21:42:56.8766700Z }, 2024-08-20T21:42:56.8766933Z { 2024-08-20T21:42:56.8767357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8767886Z "size": 32, 2024-08-20T21:42:56.8768426Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8769044Z }, 2024-08-20T21:42:56.8769275Z { 2024-08-20T21:42:56.8769687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8770234Z "size": 111, 2024-08-20T21:42:56.8770762Z "digest": "sha256:6d3ac49638ec41e1302ba5b1028a55629b0bb2cb2603212aba86d0058783ab50" 2024-08-20T21:42:56.8771493Z }, 2024-08-20T21:42:56.8771889Z { 2024-08-20T21:42:56.8772337Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8772905Z "size": 473, 2024-08-20T21:42:56.8773463Z "digest": "sha256:c5c5dbfb1dcc6e7ef2cabd2a583442e21c5b95a0cb625f0766592fff0b807d95" 2024-08-20T21:42:56.8774252Z }, 2024-08-20T21:42:56.8774501Z { 2024-08-20T21:42:56.8774949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8775495Z "size": 32, 2024-08-20T21:42:56.8776050Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8776692Z }, 2024-08-20T21:42:56.8776935Z { 2024-08-20T21:42:56.8777364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8777932Z "size": 112, 2024-08-20T21:42:56.8778498Z "digest": "sha256:327bd4a08d3c4e9ae265be2b8ce0d8ce4e4016f60e4c02a36c638789adf49ea5" 2024-08-20T21:42:56.8779138Z }, 2024-08-20T21:42:56.8779367Z { 2024-08-20T21:42:56.8779812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8780386Z "size": 566, 2024-08-20T21:42:56.8780918Z "digest": "sha256:6c520cbfb1595425e6e07ad7e2952222365e91d02cf25a4ec1f8565279b941de" 2024-08-20T21:42:56.8781555Z }, 2024-08-20T21:42:56.8781796Z { 2024-08-20T21:42:56.8782242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8782797Z "size": 43163722, 2024-08-20T21:42:56.8783369Z "digest": "sha256:a8cda4c989262a43930b0bf2507986800779f6a6ec65f8ef64bbfa72c5ca5bf6" 2024-08-20T21:42:56.8784103Z }, 2024-08-20T21:42:56.8784486Z { 2024-08-20T21:42:56.8784879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8785402Z "size": 106, 2024-08-20T21:42:56.8785916Z "digest": "sha256:3d995c530b49c1eecd803a2cfde69821f88cc0b8b5429d8955a65319217337d6" 2024-08-20T21:42:56.8786500Z }, 2024-08-20T21:42:56.8786705Z { 2024-08-20T21:42:56.8787111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8787704Z "size": 295, 2024-08-20T21:42:56.8788178Z "digest": "sha256:51635df36934a1409214ed1d1c03c997d8597887f34e547bca5fb5373e509ca8" 2024-08-20T21:42:56.8788755Z }, 2024-08-20T21:42:56.8788980Z { 2024-08-20T21:42:56.8789393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8789900Z "size": 32, 2024-08-20T21:42:56.8790409Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8790996Z }, 2024-08-20T21:42:56.8791218Z { 2024-08-20T21:42:56.8791604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8792123Z "size": 106, 2024-08-20T21:42:56.8792634Z "digest": "sha256:58f9d093cf6898c8d0fe441806e8afb6fcf8180d6ce0e61f3f8e1a5d27872b95" 2024-08-20T21:42:56.8793205Z }, 2024-08-20T21:42:56.8793431Z { 2024-08-20T21:42:56.8793830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8794346Z "size": 425, 2024-08-20T21:42:56.8794861Z "digest": "sha256:db291b17fdf0ea8442b4ee4ccf970b6bcbbf230840366e5dfe62faf43175568d" 2024-08-20T21:42:56.8795453Z }, 2024-08-20T21:42:56.8795672Z { 2024-08-20T21:42:56.8796081Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8796584Z "size": 19308700, 2024-08-20T21:42:56.8797097Z "digest": "sha256:4390932d47dc99aad011fc417cb24b8cd895f84e0247e7b1930a974848f4ae49" 2024-08-20T21:42:56.8797671Z }, 2024-08-20T21:42:56.8797876Z { 2024-08-20T21:42:56.8798277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8798797Z "size": 108, 2024-08-20T21:42:56.8799298Z "digest": "sha256:d959da1ed4a86d0902fc218b87883ea901725726db56d746621bfeb206d18ad6" 2024-08-20T21:42:56.8799861Z }, 2024-08-20T21:42:56.8800086Z { 2024-08-20T21:42:56.8800488Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8801004Z "size": 438, 2024-08-20T21:42:56.8801484Z "digest": "sha256:82769435ccd5317b20def22a2e4295f283bac4ed179599487713032c3609f189" 2024-08-20T21:42:56.8802054Z }, 2024-08-20T21:42:56.8802273Z { 2024-08-20T21:42:56.8802676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8803257Z "size": 701, 2024-08-20T21:42:56.8803786Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.8804389Z }, 2024-08-20T21:42:56.8804596Z { 2024-08-20T21:42:56.8805004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8805531Z "size": 141, 2024-08-20T21:42:56.8806040Z "digest": "sha256:50078e7792d3b05b7c50ff90f27eb679dee945052c25d2fa8590cbd90dee9ac0" 2024-08-20T21:42:56.8806691Z + exit 0 2024-08-20T21:42:56.8806929Z }, 2024-08-20T21:42:56.8807138Z { 2024-08-20T21:42:56.8807545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8808068Z "size": 135, 2024-08-20T21:42:56.8808562Z "digest": "sha256:557fee3a8871f019075f948770fb5f0860218079c7dd47fcc7982499991fb2a1" 2024-08-20T21:42:56.8809125Z }, 2024-08-20T21:42:56.8809353Z { 2024-08-20T21:42:56.8809754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8810267Z "size": 4841634229, 2024-08-20T21:42:56.8810789Z "digest": "sha256:cc0b3fc4f447d5036748372379999ebaac2ba414c77302c2910751054210e216" 2024-08-20T21:42:56.8811487Z }, 2024-08-20T21:42:56.8811893Z { 2024-08-20T21:42:56.8812318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8812882Z "size": 194, 2024-08-20T21:42:56.8813421Z "digest": "sha256:f18a0119669a20a0f6987e879f10f84bd03b89b7865214f4904cd1b9987d0b0e" 2024-08-20T21:42:56.8814037Z }, 2024-08-20T21:42:56.8814256Z { 2024-08-20T21:42:56.8814693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8815251Z "size": 1379, 2024-08-20T21:42:56.8815804Z "digest": "sha256:6197a6a036aca1f62c286b2b81941a497fa5ffed336bca2675084fe69f48f405" 2024-08-20T21:42:56.8816505Z }, 2024-08-20T21:42:56.8816747Z { 2024-08-20T21:42:56.8817187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8817736Z "size": 701, 2024-08-20T21:42:56.8818315Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.8818966Z }, 2024-08-20T21:42:56.8819212Z { 2024-08-20T21:42:56.8819631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8820196Z "size": 139, 2024-08-20T21:42:56.8820756Z "digest": "sha256:ebdeb237156e5f92ea7a5aa15d85a86b5c64ae0e6ca749609008f2e6eee128b8" 2024-08-20T21:42:56.8821406Z }, 2024-08-20T21:42:56.8821633Z { 2024-08-20T21:42:56.8822076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8822646Z "size": 3733332465, 2024-08-20T21:42:56.8823214Z "digest": "sha256:fd5d0d6ba583ac4b9de591cc4925403acd2588d10fc04114b681beb022597580" 2024-08-20T21:42:56.8823862Z }, 2024-08-20T21:42:56.8824205Z { 2024-08-20T21:42:56.8824610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8825117Z "size": 157, 2024-08-20T21:42:56.8825630Z "digest": "sha256:6b068755e588d5e5994af07fe71bd40adbbc544f6238dea5511e69862a5236dd" 2024-08-20T21:42:56.8826211Z }, 2024-08-20T21:42:56.8826437Z { 2024-08-20T21:42:56.8826826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8827358Z "size": 849, 2024-08-20T21:42:56.8827861Z "digest": "sha256:371497d14c4e377a762eec74e0c7575051758f8aa74cdc2f2436fb382df849d3" 2024-08-20T21:42:56.8828436Z }, 2024-08-20T21:42:56.8828649Z { 2024-08-20T21:42:56.8829058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8829580Z "size": 701, 2024-08-20T21:42:56.8830087Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.8830686Z }, 2024-08-20T21:42:56.8830910Z { 2024-08-20T21:42:56.8831320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8831821Z "size": 139, 2024-08-20T21:42:56.8832318Z "digest": "sha256:4b50cb25a6ac9133e2c7894845496771a2f9b23eb35a58a778bb989f9bfbac5e" 2024-08-20T21:42:56.8849554Z }, 2024-08-20T21:42:56.8849844Z { 2024-08-20T21:42:56.8850319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8850889Z "size": 32, 2024-08-20T21:42:56.8851594Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8852246Z }, 2024-08-20T21:42:56.8852491Z { 2024-08-20T21:42:56.8852918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8853491Z "size": 159, 2024-08-20T21:42:56.8854041Z "digest": "sha256:889d8139c009597aca470a64bf7aaea6d509fd8ee9a96187fa784a1d9931f350" 2024-08-20T21:42:56.8854657Z }, 2024-08-20T21:42:56.8854900Z { 2024-08-20T21:42:56.8855331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8855898Z "size": 907, 2024-08-20T21:42:56.8856450Z "digest": "sha256:2bcabd118f67bf8b2b8c6eb97e245fef2c7753e07e4ecf062b3cf6c905bdea81" 2024-08-20T21:42:56.8857097Z }, 2024-08-20T21:42:56.8857335Z { 2024-08-20T21:42:56.8857764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8858310Z "size": 701, 2024-08-20T21:42:56.8858874Z "digest": "sha256:2cebdfe4c5a8da099cc35bcc957374e8545fed39fbf41715974c4d4c1ec3efa9" 2024-08-20T21:42:56.8859519Z }, 2024-08-20T21:42:56.8859753Z { 2024-08-20T21:42:56.8860167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8860725Z "size": 134, 2024-08-20T21:42:56.8861274Z "digest": "sha256:1d831c1ea7c74b810f149c31d6eebc2d2bed6a5fb9179a9b230fec1c85f1094e" 2024-08-20T21:42:56.8861896Z }, 2024-08-20T21:42:56.8862130Z { 2024-08-20T21:42:56.8862563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8863348Z "size": 32, 2024-08-20T21:42:56.8863865Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8864494Z }, 2024-08-20T21:42:56.8864758Z { 2024-08-20T21:42:56.8865187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8865721Z "size": 158, 2024-08-20T21:42:56.8866338Z "digest": "sha256:d29d428148005386a1f6f19dd2590d9cbf58a6c1f94898e1e6ac92ea666f1641" 2024-08-20T21:42:56.8866918Z }, 2024-08-20T21:42:56.8867132Z { 2024-08-20T21:42:56.8867540Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8868070Z "size": 1579, 2024-08-20T21:42:56.8868581Z "digest": "sha256:10df3ed01753e29d9ea7d799510d1d4534671a9a4ac53db74998c1ba9c38a48c" 2024-08-20T21:42:56.8869155Z }, 2024-08-20T21:42:56.8869384Z { 2024-08-20T21:42:56.8869792Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8870319Z "size": 32, 2024-08-20T21:42:56.8870820Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8871413Z }, 2024-08-20T21:42:56.8871640Z { 2024-08-20T21:42:56.8872053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8872565Z "size": 136, 2024-08-20T21:42:56.8873064Z "digest": "sha256:3b3541d6658c782e6cf71ee8b9268b351f32109e8571447dc8993e649742d905" 2024-08-20T21:42:56.8873644Z }, 2024-08-20T21:42:56.8873851Z { 2024-08-20T21:42:56.8874260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8874786Z "size": 378, 2024-08-20T21:42:56.8875286Z "digest": "sha256:5e997601a6320f613b759c5684ae83fa39646d3c85a349797faba2e4fbc70d83" 2024-08-20T21:42:56.8875841Z }, 2024-08-20T21:42:56.8876068Z { 2024-08-20T21:42:56.8876468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8876989Z "size": 32, 2024-08-20T21:42:56.8877485Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8878080Z }, 2024-08-20T21:42:56.8878300Z { 2024-08-20T21:42:56.8878683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8879315Z "size": 104, 2024-08-20T21:42:56.8879823Z "digest": "sha256:d2924613594f110f7aa5b1681ba6bc1044cd355b5eeb17944cd11531d33a1552" 2024-08-20T21:42:56.8880405Z }, 2024-08-20T21:42:56.8880612Z { 2024-08-20T21:42:56.8881018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8881547Z "size": 1841, 2024-08-20T21:42:56.8882054Z "digest": "sha256:f66c8294ffd00da25739af287f89a292ee2b06597012e6abac6947e2197961ae" 2024-08-20T21:42:56.8882614Z }, 2024-08-20T21:42:56.8882833Z { 2024-08-20T21:42:56.8883235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8883755Z "size": 7529594, 2024-08-20T21:42:56.8884271Z "digest": "sha256:9da0ec93b0d7d37059844ca82dd8b0ce33bdd51614cc9bd66c391ffe64094ac7" 2024-08-20T21:42:56.8884865Z }, 2024-08-20T21:42:56.8885084Z { 2024-08-20T21:42:56.8885469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8885988Z "size": 106, 2024-08-20T21:42:56.8886484Z "digest": "sha256:d476e4b8627568a99c85e93d60a7ce5676a42e9ee10a5b1054348658d77f56f3" 2024-08-20T21:42:56.8887056Z }, 2024-08-20T21:42:56.8887265Z { 2024-08-20T21:42:56.8887666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8888186Z "size": 163, 2024-08-20T21:42:56.8888683Z "digest": "sha256:397c513e7e07f5c83f0d0c8c407a7bdc3e7167a57744262cd5a1282ee451d52d" 2024-08-20T21:42:56.8889244Z }, 2024-08-20T21:42:56.8889464Z { 2024-08-20T21:42:56.8889865Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8890373Z "size": 7943, 2024-08-20T21:42:56.8890893Z "digest": "sha256:5d6d58e6432b6e8048d3f6cccb787a1cfe9d1203f742a9f0fddfb00338cccae7" 2024-08-20T21:42:56.8891999Z }, 2024-08-20T21:42:56.8892240Z { 2024-08-20T21:42:56.8892657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8893221Z "size": 8066, 2024-08-20T21:42:56.8893793Z "digest": "sha256:f9db7741af1464b9e28a0bab667a6b4559ab7f3d319add7a50bc1dcc2b1e97fe" 2024-08-20T21:42:56.8894441Z }, 2024-08-20T21:42:56.8894660Z { 2024-08-20T21:42:56.8895089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8895647Z "size": 303, 2024-08-20T21:42:56.8896185Z "digest": "sha256:c0f5c95392e3593f2bccab197736341bfef11427975492f581c088b8e21401a1" 2024-08-20T21:42:56.8896791Z }, 2024-08-20T21:42:56.8897022Z { 2024-08-20T21:42:56.8897451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8897990Z "size": 32, 2024-08-20T21:42:56.8898532Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8899163Z }, 2024-08-20T21:42:56.8899401Z { 2024-08-20T21:42:56.8899815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8900371Z "size": 108, 2024-08-20T21:42:56.8900920Z "digest": "sha256:d6d4b368f5e67517827a02d0e324ced915843fa69780cbf87cac205ffcf04032" 2024-08-20T21:42:56.8901536Z }, 2024-08-20T21:42:56.8901754Z { 2024-08-20T21:42:56.8902185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8902746Z "size": 54145664, 2024-08-20T21:42:56.8903298Z "digest": "sha256:6e341d8be4cd61d3439f2becbc390d9d27179704631d29225ba4c01bc518ff56" 2024-08-20T21:42:56.8904034Z }, 2024-08-20T21:42:56.8904250Z { 2024-08-20T21:42:56.8904644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:42:56.8905144Z "size": 32, 2024-08-20T21:42:56.8905646Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:42:56.8906229Z } 2024-08-20T21:42:56.8906446Z ] 2024-08-20T21:42:56.8906646Z } 2024-08-20T21:42:56.9005144Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:42:56.9005578Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:42:56.9006084Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T21:42:56.9012600Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.9013101Z env: 2024-08-20T21:42:56.9013369Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.9014492Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.9015638Z ##[endgroup] 2024-08-20T21:42:56.9043574Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.9094509Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T21:42:56.9095086Z with: 2024-08-20T21:42:56.9096122Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.9097412Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.9097927Z env: 2024-08-20T21:42:56.9098185Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.9098514Z ##[endgroup] 2024-08-20T21:42:56.9113651Z ##[group]Run set -x 2024-08-20T21:42:56.9113960Z set -x 2024-08-20T21:42:56.9114231Z set +e 2024-08-20T21:42:56.9114491Z  2024-08-20T21:42:56.9114722Z login() { 2024-08-20T21:42:56.9115341Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:42:56.9116013Z } 2024-08-20T21:42:56.9116258Z  2024-08-20T21:42:56.9116547Z retry () { 2024-08-20T21:42:56.9116901Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:42:56.9117296Z } 2024-08-20T21:42:56.9117538Z  2024-08-20T21:42:56.9117818Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:42:56.9118285Z  2024-08-20T21:42:56.9118524Z set -e 2024-08-20T21:42:56.9118955Z # ignore output since only exit code is used for conditional 2024-08-20T21:42:56.9119607Z # only pull docker image if it's not available locally 2024-08-20T21:42:56.9120310Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T21:42:56.9120960Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T21:42:56.9121361Z fi 2024-08-20T21:42:56.9126438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:42:56.9126886Z env: 2024-08-20T21:42:56.9127135Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:42:56.9128171Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:56.9129351Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.9129837Z ##[endgroup] 2024-08-20T21:42:56.9152771Z + set +e 2024-08-20T21:42:56.9153397Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.9154038Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:56.9155887Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:42:56.9157122Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:42:57.4179235Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:42:57.4180051Z Configure a credential helper to remove this warning. See 2024-08-20T21:42:57.4180899Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:42:57.4181434Z 2024-08-20T21:42:57.4181569Z Login Succeeded 2024-08-20T21:42:57.4193051Z + set -e 2024-08-20T21:42:57.4194427Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.4320314Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.4322420Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:42:57.6925789Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks 2024-08-20T21:42:57.6929106Z 762bedf4b1b7: Pulling fs layer 2024-08-20T21:42:57.6929514Z 347beb78d39b: Pulling fs layer 2024-08-20T21:42:57.6929917Z 71b614615bcb: Pulling fs layer 2024-08-20T21:42:57.6930599Z f96c8558662d: Pulling fs layer 2024-08-20T21:42:57.6931009Z c81ff9d20d36: Pulling fs layer 2024-08-20T21:42:57.6931553Z 151e9a9d2894: Pulling fs layer 2024-08-20T21:42:57.6931912Z 8538dcd89f0c: Pulling fs layer 2024-08-20T21:42:57.6932313Z cd13007f6f09: Pulling fs layer 2024-08-20T21:42:57.6932692Z 74fd96b6ebd9: Pulling fs layer 2024-08-20T21:42:57.6933074Z ce4e1800db98: Pulling fs layer 2024-08-20T21:42:57.6933433Z 402b067f30bf: Pulling fs layer 2024-08-20T21:42:57.6933808Z 2cebdfe4c5a8: Pulling fs layer 2024-08-20T21:42:57.6934171Z c81ff9d20d36: Waiting 2024-08-20T21:42:57.6934537Z 151e9a9d2894: Waiting 2024-08-20T21:42:57.6934857Z c6e13ff8aa9a: Pulling fs layer 2024-08-20T21:42:57.6935222Z 8538dcd89f0c: Waiting 2024-08-20T21:42:57.6935555Z 4f4fb700ef54: Pulling fs layer 2024-08-20T21:42:57.6935936Z e238df05ddcc: Pulling fs layer 2024-08-20T21:42:57.6936280Z cd13007f6f09: Waiting 2024-08-20T21:42:57.6936613Z a88675bdc723: Pulling fs layer 2024-08-20T21:42:57.6936993Z f4cb80c411d1: Pulling fs layer 2024-08-20T21:42:57.6937358Z c9589ebd9ae7: Pulling fs layer 2024-08-20T21:42:57.6937740Z dd51c681db14: Pulling fs layer 2024-08-20T21:42:57.6938110Z fc983e33a069: Pulling fs layer 2024-08-20T21:42:57.6938479Z bc705d688c77: Pulling fs layer 2024-08-20T21:42:57.6938994Z 99f6c1ac7582: Pulling fs layer 2024-08-20T21:42:57.6939366Z c409d27a5a95: Pulling fs layer 2024-08-20T21:42:57.6939737Z a3a398ca1120: Pulling fs layer 2024-08-20T21:42:57.6940096Z 74fd96b6ebd9: Waiting 2024-08-20T21:42:57.6940407Z 314875f5fae4: Pulling fs layer 2024-08-20T21:42:57.6940763Z ce4e1800db98: Waiting 2024-08-20T21:42:57.6941097Z 8a5e90dafefa: Pulling fs layer 2024-08-20T21:42:57.6941445Z 402b067f30bf: Waiting 2024-08-20T21:42:57.6941773Z e3b946ab3378: Pulling fs layer 2024-08-20T21:42:57.6942135Z 2cebdfe4c5a8: Waiting 2024-08-20T21:42:57.6942469Z 14a85a990f71: Pulling fs layer 2024-08-20T21:42:57.6942812Z c6e13ff8aa9a: Waiting 2024-08-20T21:42:57.6943147Z 16d809b167d4: Pulling fs layer 2024-08-20T21:42:57.6943525Z 9b1a95eedb83: Pulling fs layer 2024-08-20T21:42:57.6943905Z b35ca0542723: Pulling fs layer 2024-08-20T21:42:57.6944241Z 4f4fb700ef54: Waiting 2024-08-20T21:42:57.6944561Z e238df05ddcc: Waiting 2024-08-20T21:42:57.6944895Z bad10ac3a627: Pulling fs layer 2024-08-20T21:42:57.6945482Z 943da2164c7a: Pulling fs layer 2024-08-20T21:42:57.6945861Z 2d276441bd8a: Pulling fs layer 2024-08-20T21:42:57.6946286Z f96c8558662d: Waiting 2024-08-20T21:42:57.6946622Z a3ef8d7d02be: Pulling fs layer 2024-08-20T21:42:57.6946988Z 6aaed7a78749: Pulling fs layer 2024-08-20T21:42:57.6947362Z 944e78083916: Pulling fs layer 2024-08-20T21:42:57.6947718Z 314875f5fae4: Waiting 2024-08-20T21:42:57.6948049Z 9602de36240c: Pulling fs layer 2024-08-20T21:42:57.6948395Z 8a5e90dafefa: Waiting 2024-08-20T21:42:57.6948713Z f4cb80c411d1: Waiting 2024-08-20T21:42:57.6949048Z 6d3ac49638ec: Pulling fs layer 2024-08-20T21:42:57.6949394Z a88675bdc723: Waiting 2024-08-20T21:42:57.6949705Z bc705d688c77: Waiting 2024-08-20T21:42:57.6950022Z c9589ebd9ae7: Waiting 2024-08-20T21:42:57.6950369Z c5c5dbfb1dcc: Pulling fs layer 2024-08-20T21:42:57.6950719Z e3b946ab3378: Waiting 2024-08-20T21:42:57.6951042Z c409d27a5a95: Waiting 2024-08-20T21:42:57.6951365Z 99f6c1ac7582: Waiting 2024-08-20T21:42:57.6951674Z dd51c681db14: Waiting 2024-08-20T21:42:57.6952001Z fc983e33a069: Waiting 2024-08-20T21:42:57.6952318Z 14a85a990f71: Waiting 2024-08-20T21:42:57.6952637Z a3a398ca1120: Waiting 2024-08-20T21:42:57.6952955Z 327bd4a08d3c: Pulling fs layer 2024-08-20T21:42:57.6953346Z 6c520cbfb159: Pulling fs layer 2024-08-20T21:42:57.6953712Z 6aaed7a78749: Waiting 2024-08-20T21:42:57.6954091Z a8cda4c98926: Pulling fs layer 2024-08-20T21:42:57.6954466Z a3ef8d7d02be: Waiting 2024-08-20T21:42:57.6954769Z 9b1a95eedb83: Waiting 2024-08-20T21:42:57.6955083Z 9602de36240c: Waiting 2024-08-20T21:42:57.6955472Z 944e78083916: Waiting 2024-08-20T21:42:57.6955786Z 3d995c530b49: Pulling fs layer 2024-08-20T21:42:57.6956158Z bad10ac3a627: Waiting 2024-08-20T21:42:57.6956477Z 16d809b167d4: Waiting 2024-08-20T21:42:57.6956973Z 51635df36934: Pulling fs layer 2024-08-20T21:42:57.6957325Z c5c5dbfb1dcc: Waiting 2024-08-20T21:42:57.6957646Z 327bd4a08d3c: Waiting 2024-08-20T21:42:57.6957963Z b35ca0542723: Waiting 2024-08-20T21:42:57.6958297Z 58f9d093cf68: Pulling fs layer 2024-08-20T21:42:57.6958648Z 943da2164c7a: Waiting 2024-08-20T21:42:57.6958964Z 3d995c530b49: Waiting 2024-08-20T21:42:57.6959287Z db291b17fdf0: Pulling fs layer 2024-08-20T21:42:57.6959635Z 6c520cbfb159: Waiting 2024-08-20T21:42:57.6959948Z 51635df36934: Waiting 2024-08-20T21:42:57.6960271Z 4390932d47dc: Pulling fs layer 2024-08-20T21:42:57.6960674Z 6d3ac49638ec: Waiting 2024-08-20T21:42:57.6960983Z d959da1ed4a8: Pulling fs layer 2024-08-20T21:42:57.6961342Z a8cda4c98926: Waiting 2024-08-20T21:42:57.6961649Z 4390932d47dc: Waiting 2024-08-20T21:42:57.6961957Z 82769435ccd5: Pulling fs layer 2024-08-20T21:42:57.6962313Z d959da1ed4a8: Waiting 2024-08-20T21:42:57.6962641Z 50078e7792d3: Pulling fs layer 2024-08-20T21:42:57.6963014Z 557fee3a8871: Pulling fs layer 2024-08-20T21:42:57.6963376Z cc0b3fc4f447: Pulling fs layer 2024-08-20T21:42:57.6963751Z f18a0119669a: Pulling fs layer 2024-08-20T21:42:57.6964128Z 6197a6a036ac: Pulling fs layer 2024-08-20T21:42:57.6964500Z ebdeb237156e: Pulling fs layer 2024-08-20T21:42:57.6964949Z 6197a6a036ac: Waiting 2024-08-20T21:42:57.6965281Z fd5d0d6ba583: Pulling fs layer 2024-08-20T21:42:57.6965642Z 82769435ccd5: Waiting 2024-08-20T21:42:57.6965951Z 6b068755e588: Pulling fs layer 2024-08-20T21:42:57.6966313Z 557fee3a8871: Waiting 2024-08-20T21:42:57.6966632Z cc0b3fc4f447: Waiting 2024-08-20T21:42:57.6966961Z 371497d14c4e: Pulling fs layer 2024-08-20T21:42:57.6967299Z f18a0119669a: Waiting 2024-08-20T21:42:57.6967610Z ebdeb237156e: Waiting 2024-08-20T21:42:57.6967921Z 50078e7792d3: Waiting 2024-08-20T21:42:57.6968249Z 4b50cb25a6ac: Pulling fs layer 2024-08-20T21:42:57.6968607Z 889d8139c009: Pulling fs layer 2024-08-20T21:42:57.6968978Z 2bcabd118f67: Pulling fs layer 2024-08-20T21:42:57.6969354Z 1d831c1ea7c7: Pulling fs layer 2024-08-20T21:42:57.6969712Z d29d42814800: Pulling fs layer 2024-08-20T21:42:57.6970089Z 10df3ed01753: Pulling fs layer 2024-08-20T21:42:57.6970461Z 3b3541d6658c: Pulling fs layer 2024-08-20T21:42:57.6970818Z 10df3ed01753: Waiting 2024-08-20T21:42:57.6971118Z d29d42814800: Waiting 2024-08-20T21:42:57.6971589Z 5e997601a632: Pulling fs layer 2024-08-20T21:42:57.6971950Z 2bcabd118f67: Waiting 2024-08-20T21:42:57.6972268Z 1d831c1ea7c7: Waiting 2024-08-20T21:42:57.6972579Z d2924613594f: Pulling fs layer 2024-08-20T21:42:57.6972934Z 5e997601a632: Waiting 2024-08-20T21:42:57.6973264Z f66c8294ffd0: Pulling fs layer 2024-08-20T21:42:57.6973628Z 9da0ec93b0d7: Pulling fs layer 2024-08-20T21:42:57.6973999Z f66c8294ffd0: Waiting 2024-08-20T21:42:57.6974325Z d476e4b86275: Pulling fs layer 2024-08-20T21:42:57.6974695Z 397c513e7e07: Pulling fs layer 2024-08-20T21:42:57.6975047Z 5d6d58e6432b: Pulling fs layer 2024-08-20T21:42:57.6975407Z 9da0ec93b0d7: Waiting 2024-08-20T21:42:57.6975739Z f9db7741af14: Pulling fs layer 2024-08-20T21:42:57.6976100Z c0f5c95392e3: Pulling fs layer 2024-08-20T21:42:57.6976457Z 5d6d58e6432b: Waiting 2024-08-20T21:42:57.6976769Z d476e4b86275: Waiting 2024-08-20T21:42:57.6977077Z 397c513e7e07: Waiting 2024-08-20T21:42:57.6977387Z d6d4b368f5e6: Pulling fs layer 2024-08-20T21:42:57.6977751Z c0f5c95392e3: Waiting 2024-08-20T21:42:57.6978077Z 6e341d8be4cd: Pulling fs layer 2024-08-20T21:42:57.6978440Z 6e341d8be4cd: Waiting 2024-08-20T21:42:57.7610747Z 347beb78d39b: Download complete 2024-08-20T21:42:57.8453425Z f96c8558662d: Verifying Checksum 2024-08-20T21:42:57.8454610Z f96c8558662d: Download complete 2024-08-20T21:42:57.9035158Z c81ff9d20d36: Verifying Checksum 2024-08-20T21:42:57.9035643Z c81ff9d20d36: Download complete 2024-08-20T21:42:57.9738415Z 151e9a9d2894: Verifying Checksum 2024-08-20T21:42:57.9739059Z 151e9a9d2894: Download complete 2024-08-20T21:42:58.0415865Z 8538dcd89f0c: Download complete 2024-08-20T21:42:58.0465525Z 762bedf4b1b7: Verifying Checksum 2024-08-20T21:42:58.0466678Z 762bedf4b1b7: Download complete 2024-08-20T21:42:58.1081968Z cd13007f6f09: Verifying Checksum 2024-08-20T21:42:58.1082445Z cd13007f6f09: Download complete 2024-08-20T21:42:58.1815775Z ce4e1800db98: Verifying Checksum 2024-08-20T21:42:58.1816245Z ce4e1800db98: Download complete 2024-08-20T21:42:58.2524416Z 402b067f30bf: Verifying Checksum 2024-08-20T21:42:58.2524876Z 402b067f30bf: Download complete 2024-08-20T21:42:58.3173234Z 2cebdfe4c5a8: Verifying Checksum 2024-08-20T21:42:58.3173712Z 2cebdfe4c5a8: Download complete 2024-08-20T21:42:58.7498370Z 762bedf4b1b7: Pull complete 2024-08-20T21:42:58.7650439Z 347beb78d39b: Pull complete 2024-08-20T21:42:59.2157369Z 74fd96b6ebd9: Verifying Checksum 2024-08-20T21:42:59.2157846Z 74fd96b6ebd9: Download complete 2024-08-20T21:42:59.2282019Z 4f4fb700ef54: Verifying Checksum 2024-08-20T21:42:59.2282726Z 4f4fb700ef54: Download complete 2024-08-20T21:42:59.3080064Z e238df05ddcc: Verifying Checksum 2024-08-20T21:42:59.3080771Z e238df05ddcc: Download complete 2024-08-20T21:42:59.3929296Z a88675bdc723: Verifying Checksum 2024-08-20T21:42:59.3929856Z a88675bdc723: Download complete 2024-08-20T21:42:59.4684353Z f4cb80c411d1: Verifying Checksum 2024-08-20T21:42:59.4684876Z f4cb80c411d1: Download complete 2024-08-20T21:42:59.5578035Z c9589ebd9ae7: Verifying Checksum 2024-08-20T21:42:59.5578718Z c9589ebd9ae7: Download complete 2024-08-20T21:42:59.6258194Z dd51c681db14: Download complete 2024-08-20T21:42:59.6990875Z fc983e33a069: Verifying Checksum 2024-08-20T21:42:59.6991565Z fc983e33a069: Download complete 2024-08-20T21:42:59.7735277Z bc705d688c77: Verifying Checksum 2024-08-20T21:42:59.7736495Z bc705d688c77: Download complete 2024-08-20T21:42:59.8550150Z 99f6c1ac7582: Verifying Checksum 2024-08-20T21:42:59.8551003Z 99f6c1ac7582: Download complete 2024-08-20T21:42:59.9266529Z c409d27a5a95: Verifying Checksum 2024-08-20T21:42:59.9267229Z c409d27a5a95: Download complete 2024-08-20T21:43:00.8803036Z 71b614615bcb: Verifying Checksum 2024-08-20T21:43:00.8803523Z 71b614615bcb: Download complete 2024-08-20T21:43:00.9568120Z 314875f5fae4: Download complete 2024-08-20T21:43:01.0017075Z a3a398ca1120: Verifying Checksum 2024-08-20T21:43:01.0017476Z a3a398ca1120: Download complete 2024-08-20T21:43:01.0423972Z 8a5e90dafefa: Download complete 2024-08-20T21:43:01.0702321Z e3b946ab3378: Download complete 2024-08-20T21:43:01.1096898Z 14a85a990f71: Verifying Checksum 2024-08-20T21:43:01.1097402Z 14a85a990f71: Download complete 2024-08-20T21:43:01.1429909Z 16d809b167d4: Verifying Checksum 2024-08-20T21:43:01.1430377Z 16d809b167d4: Download complete 2024-08-20T21:43:01.2101287Z b35ca0542723: Download complete 2024-08-20T21:43:01.2923130Z bad10ac3a627: Verifying Checksum 2024-08-20T21:43:01.2923592Z bad10ac3a627: Download complete 2024-08-20T21:43:01.3684779Z 943da2164c7a: Download complete 2024-08-20T21:43:01.4342766Z 2d276441bd8a: Verifying Checksum 2024-08-20T21:43:01.4343238Z 2d276441bd8a: Download complete 2024-08-20T21:43:01.5157854Z a3ef8d7d02be: Verifying Checksum 2024-08-20T21:43:01.5158308Z a3ef8d7d02be: Download complete 2024-08-20T21:43:01.5887046Z 6aaed7a78749: Verifying Checksum 2024-08-20T21:43:01.5887510Z 6aaed7a78749: Download complete 2024-08-20T21:43:01.6563851Z 944e78083916: Verifying Checksum 2024-08-20T21:43:01.6564318Z 944e78083916: Download complete 2024-08-20T21:43:01.7334152Z 9602de36240c: Verifying Checksum 2024-08-20T21:43:01.7334629Z 9602de36240c: Download complete 2024-08-20T21:43:01.8032375Z 6d3ac49638ec: Verifying Checksum 2024-08-20T21:43:01.8032848Z 6d3ac49638ec: Download complete 2024-08-20T21:43:01.8825898Z c5c5dbfb1dcc: Verifying Checksum 2024-08-20T21:43:01.8826357Z c5c5dbfb1dcc: Download complete 2024-08-20T21:43:01.9506956Z 327bd4a08d3c: Verifying Checksum 2024-08-20T21:43:01.9507472Z 327bd4a08d3c: Download complete 2024-08-20T21:43:02.0156771Z 6c520cbfb159: Verifying Checksum 2024-08-20T21:43:02.0157213Z 6c520cbfb159: Download complete 2024-08-20T21:43:02.5100531Z a8cda4c98926: Verifying Checksum 2024-08-20T21:43:02.5100968Z a8cda4c98926: Download complete 2024-08-20T21:43:02.5779663Z 3d995c530b49: Download complete 2024-08-20T21:43:02.6523151Z 51635df36934: Verifying Checksum 2024-08-20T21:43:02.6523649Z 51635df36934: Download complete 2024-08-20T21:43:02.7204761Z 58f9d093cf68: Download complete 2024-08-20T21:43:02.8153172Z db291b17fdf0: Download complete 2024-08-20T21:43:03.0537538Z 4390932d47dc: Verifying Checksum 2024-08-20T21:43:03.0538033Z 4390932d47dc: Download complete 2024-08-20T21:43:03.1235631Z d959da1ed4a8: Verifying Checksum 2024-08-20T21:43:03.1236106Z d959da1ed4a8: Download complete 2024-08-20T21:43:03.1825713Z 82769435ccd5: Verifying Checksum 2024-08-20T21:43:03.1826181Z 82769435ccd5: Download complete 2024-08-20T21:43:03.2428762Z 50078e7792d3: Verifying Checksum 2024-08-20T21:43:03.2429219Z 50078e7792d3: Download complete 2024-08-20T21:43:03.3252871Z 557fee3a8871: Verifying Checksum 2024-08-20T21:43:03.3253400Z 557fee3a8871: Download complete 2024-08-20T21:43:07.2780318Z 9b1a95eedb83: Verifying Checksum 2024-08-20T21:43:07.2780806Z 9b1a95eedb83: Download complete 2024-08-20T21:43:07.3627425Z f18a0119669a: Verifying Checksum 2024-08-20T21:43:07.3627841Z f18a0119669a: Download complete 2024-08-20T21:43:07.4240437Z 6197a6a036ac: Download complete 2024-08-20T21:43:07.4970338Z ebdeb237156e: Verifying Checksum 2024-08-20T21:43:07.4971124Z ebdeb237156e: Download complete 2024-08-20T21:43:07.7387604Z 71b614615bcb: Pull complete 2024-08-20T21:43:08.1696838Z f96c8558662d: Pull complete 2024-08-20T21:43:08.6457581Z c81ff9d20d36: Pull complete 2024-08-20T21:43:09.1253601Z 151e9a9d2894: Pull complete 2024-08-20T21:43:09.5580590Z 8538dcd89f0c: Pull complete 2024-08-20T21:43:10.0730124Z cd13007f6f09: Pull complete 2024-08-20T21:43:12.5733359Z 74fd96b6ebd9: Pull complete 2024-08-20T21:43:13.0529419Z ce4e1800db98: Pull complete 2024-08-20T21:43:13.5173258Z 402b067f30bf: Pull complete 2024-08-20T21:43:13.9770025Z 2cebdfe4c5a8: Pull complete 2024-08-20T21:43:24.9994781Z c6e13ff8aa9a: Verifying Checksum 2024-08-20T21:43:24.9995378Z c6e13ff8aa9a: Download complete 2024-08-20T21:43:25.1591200Z 371497d14c4e: Download complete 2024-08-20T21:43:25.2235155Z 4b50cb25a6ac: Verifying Checksum 2024-08-20T21:43:25.2235671Z 4b50cb25a6ac: Download complete 2024-08-20T21:43:25.3300428Z 889d8139c009: Verifying Checksum 2024-08-20T21:43:25.3300949Z 889d8139c009: Download complete 2024-08-20T21:43:25.4039157Z 2bcabd118f67: Verifying Checksum 2024-08-20T21:43:25.4039790Z 2bcabd118f67: Download complete 2024-08-20T21:43:25.4819178Z 1d831c1ea7c7: Download complete 2024-08-20T21:43:25.5528048Z d29d42814800: Verifying Checksum 2024-08-20T21:43:25.5528502Z d29d42814800: Download complete 2024-08-20T21:43:25.6289483Z 10df3ed01753: Verifying Checksum 2024-08-20T21:43:25.6289954Z 10df3ed01753: Download complete 2024-08-20T21:43:25.6974588Z 3b3541d6658c: Verifying Checksum 2024-08-20T21:43:25.6975036Z 3b3541d6658c: Download complete 2024-08-20T21:43:25.7637987Z 5e997601a632: Verifying Checksum 2024-08-20T21:43:25.7638452Z 5e997601a632: Download complete 2024-08-20T21:43:25.8312414Z d2924613594f: Verifying Checksum 2024-08-20T21:43:25.8312899Z d2924613594f: Download complete 2024-08-20T21:43:25.9020758Z f66c8294ffd0: Download complete 2024-08-20T21:43:26.0237518Z 9da0ec93b0d7: Verifying Checksum 2024-08-20T21:43:26.0238305Z 9da0ec93b0d7: Download complete 2024-08-20T21:43:26.1016582Z d476e4b86275: Verifying Checksum 2024-08-20T21:43:26.1017029Z d476e4b86275: Download complete 2024-08-20T21:43:26.1885001Z 397c513e7e07: Download complete 2024-08-20T21:43:26.2626423Z 5d6d58e6432b: Verifying Checksum 2024-08-20T21:43:26.2626952Z 5d6d58e6432b: Download complete 2024-08-20T21:43:26.3247141Z f9db7741af14: Verifying Checksum 2024-08-20T21:43:26.3247630Z f9db7741af14: Download complete 2024-08-20T21:43:26.3947345Z c0f5c95392e3: Download complete 2024-08-20T21:43:26.4652692Z d6d4b368f5e6: Verifying Checksum 2024-08-20T21:43:26.4653148Z d6d4b368f5e6: Download complete 2024-08-20T21:43:27.0813745Z 6e341d8be4cd: Verifying Checksum 2024-08-20T21:43:27.0814217Z 6e341d8be4cd: Download complete 2024-08-20T21:43:44.9693653Z fd5d0d6ba583: Verifying Checksum 2024-08-20T21:43:44.9694377Z fd5d0d6ba583: Download complete 2024-08-20T21:43:51.7939849Z cc0b3fc4f447: Verifying Checksum 2024-08-20T21:43:51.7940322Z cc0b3fc4f447: Download complete 2024-08-20T21:44:02.8725320Z c6e13ff8aa9a: Pull complete 2024-08-20T21:44:03.3596496Z 4f4fb700ef54: Pull complete 2024-08-20T21:44:03.6163029Z e238df05ddcc: Pull complete 2024-08-20T21:44:04.0857422Z a88675bdc723: Pull complete 2024-08-20T21:44:04.5865402Z f4cb80c411d1: Pull complete 2024-08-20T21:44:05.1234944Z c9589ebd9ae7: Pull complete 2024-08-20T21:44:05.5735786Z dd51c681db14: Pull complete 2024-08-20T21:44:06.0269185Z fc983e33a069: Pull complete 2024-08-20T21:44:06.4919425Z bc705d688c77: Pull complete 2024-08-20T21:44:07.0046351Z 99f6c1ac7582: Pull complete 2024-08-20T21:44:07.3023874Z c409d27a5a95: Pull complete 2024-08-20T21:44:09.2774206Z a3a398ca1120: Pull complete 2024-08-20T21:44:09.7293271Z 314875f5fae4: Pull complete 2024-08-20T21:44:10.1311304Z 8a5e90dafefa: Pull complete 2024-08-20T21:44:10.5514421Z e3b946ab3378: Pull complete 2024-08-20T21:44:11.0070598Z 14a85a990f71: Pull complete 2024-08-20T21:44:11.4639470Z 16d809b167d4: Pull complete 2024-08-20T21:44:19.1117697Z 9b1a95eedb83: Pull complete 2024-08-20T21:44:19.5443786Z b35ca0542723: Pull complete 2024-08-20T21:44:19.9994695Z bad10ac3a627: Pull complete 2024-08-20T21:44:20.3521081Z 943da2164c7a: Pull complete 2024-08-20T21:44:20.5786510Z 2d276441bd8a: Pull complete 2024-08-20T21:44:21.3416631Z a3ef8d7d02be: Pull complete 2024-08-20T21:44:21.6785618Z 6aaed7a78749: Pull complete 2024-08-20T21:44:21.9153292Z 944e78083916: Pull complete 2024-08-20T21:44:22.2232392Z 9602de36240c: Pull complete 2024-08-20T21:44:22.7818130Z 6d3ac49638ec: Pull complete 2024-08-20T21:44:22.8670582Z c5c5dbfb1dcc: Pull complete 2024-08-20T21:44:23.5934804Z 327bd4a08d3c: Pull complete 2024-08-20T21:44:23.9378663Z 6c520cbfb159: Pull complete 2024-08-20T21:44:25.9998561Z a8cda4c98926: Pull complete 2024-08-20T21:44:26.0424467Z 3d995c530b49: Pull complete 2024-08-20T21:44:26.0773365Z 51635df36934: Pull complete 2024-08-20T21:44:26.1729293Z 58f9d093cf68: Pull complete 2024-08-20T21:44:26.2233963Z db291b17fdf0: Pull complete 2024-08-20T21:44:26.5286375Z 4390932d47dc: Pull complete 2024-08-20T21:44:26.7448326Z d959da1ed4a8: Pull complete 2024-08-20T21:44:27.1656533Z 82769435ccd5: Pull complete 2024-08-20T21:44:27.8951046Z 50078e7792d3: Pull complete 2024-08-20T21:44:28.2279623Z 557fee3a8871: Pull complete 2024-08-20T21:44:57.0173501Z cc0b3fc4f447: Pull complete 2024-08-20T21:44:57.4542385Z f18a0119669a: Pull complete 2024-08-20T21:44:57.7757006Z 6197a6a036ac: Pull complete 2024-08-20T21:44:58.4572443Z ebdeb237156e: Pull complete 2024-08-20T21:45:45.4810122Z fd5d0d6ba583: Pull complete 2024-08-20T21:45:45.9249761Z 6b068755e588: Pull complete 2024-08-20T21:45:46.3712275Z 371497d14c4e: Pull complete 2024-08-20T21:45:47.1356077Z 4b50cb25a6ac: Pull complete 2024-08-20T21:45:47.9763977Z 889d8139c009: Pull complete 2024-08-20T21:45:48.4475467Z 2bcabd118f67: Pull complete 2024-08-20T21:45:49.3104658Z 1d831c1ea7c7: Pull complete 2024-08-20T21:45:50.1925870Z d29d42814800: Pull complete 2024-08-20T21:45:50.6513887Z 10df3ed01753: Pull complete 2024-08-20T21:45:51.2882096Z 3b3541d6658c: Pull complete 2024-08-20T21:45:51.6703666Z 5e997601a632: Pull complete 2024-08-20T21:45:52.5455703Z d2924613594f: Pull complete 2024-08-20T21:45:52.9402863Z f66c8294ffd0: Pull complete 2024-08-20T21:45:53.5013211Z 9da0ec93b0d7: Pull complete 2024-08-20T21:45:53.9694313Z d476e4b86275: Pull complete 2024-08-20T21:45:54.4211495Z 397c513e7e07: Pull complete 2024-08-20T21:45:54.7185583Z 5d6d58e6432b: Pull complete 2024-08-20T21:45:55.0564360Z f9db7741af14: Pull complete 2024-08-20T21:45:55.4833576Z c0f5c95392e3: Pull complete 2024-08-20T21:45:56.1769589Z d6d4b368f5e6: Pull complete 2024-08-20T21:45:57.7923260Z 6e341d8be4cd: Pull complete 2024-08-20T21:45:58.3828990Z Digest: sha256:ed18f711327455e269271a17448384275684348a472ba289f00904bd14501fa9 2024-08-20T21:45:58.4668647Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:45:58.5113661Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:45:58.5230285Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:45:58.5231249Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:45:58.5239404Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:45:58.5239863Z env: 2024-08-20T21:45:58.5240123Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:58.5240448Z ##[endgroup] 2024-08-20T21:45:58.5488510Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:45:58.5489280Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:45:58.5489967Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T21:45:58.5490622Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:45:58.5496956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:45:58.5497469Z env: 2024-08-20T21:45:58.5497933Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:58.5498281Z ##[endgroup] 2024-08-20T21:45:58.7893653Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:45:58.8053485Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T21:45:58.8058429Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T21:45:58.9214566Z Prepare all required actions 2024-08-20T21:45:58.9215074Z Getting action download info 2024-08-20T21:45:59.0225215Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T21:45:59.2215599Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T21:45:59.3511670Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T21:45:59.3512245Z with: 2024-08-20T21:45:59.3512527Z name: linux-jammy-py3.8-gcc11-build 2024-08-20T21:45:59.3512920Z s3-bucket: gha-artifacts 2024-08-20T21:45:59.3513228Z env: 2024-08-20T21:45:59.3513456Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:59.3513770Z ##[endgroup] 2024-08-20T21:45:59.3752091Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:45:59.3752512Z with: 2024-08-20T21:45:59.3752789Z name: linux-jammy-py3.8-gcc11-build 2024-08-20T21:45:59.3753178Z s3-bucket: gha-artifacts 2024-08-20T21:45:59.3753543Z region: us-east-1 2024-08-20T21:45:59.3753802Z env: 2024-08-20T21:45:59.3754049Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:45:59.3754363Z ##[endgroup] 2024-08-20T21:45:59.8397140Z (node:485118) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:45:59.8397867Z 2024-08-20T21:45:59.8398136Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:45:59.8398866Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:45:59.8399788Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:45:59.9170343Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-jammy-py3.8-gcc11-build/ 2024-08-20T21:45:59.9171557Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:46:03.6590617Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:46:03.6596170Z Artifact download has finished successfully 2024-08-20T21:46:03.6832752Z ##[group]Run unzip -o artifacts.zip 2024-08-20T21:46:03.6833171Z unzip -o artifacts.zip 2024-08-20T21:46:03.6838831Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:03.6839286Z env: 2024-08-20T21:46:03.6839523Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:03.6839837Z ##[endgroup] 2024-08-20T21:46:03.6893096Z Archive: artifacts.zip 2024-08-20T21:46:03.6893952Z creating: dist/ 2024-08-20T21:46:04.6160733Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-20T21:46:04.6161482Z creating: build/custom_test_artifacts/ 2024-08-20T21:46:04.6162089Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T21:46:04.6162810Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T21:46:04.6163665Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:04.6164699Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:04.6184906Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:04.6186125Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:04.6187177Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:04.6188187Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:04.6189598Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:04.6190772Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:04.6191812Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:04.6192850Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:04.6194182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:04.6195409Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:04.6196607Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:04.6197807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:04.6199042Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:04.6200267Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:04.6201302Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:04.6202252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:04.6203218Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T21:46:04.6204283Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T21:46:04.6205487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T21:46:04.6206649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T21:46:04.6207732Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T21:46:04.6208859Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T21:46:04.6209978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T21:46:04.6211221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T21:46:04.6212419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T21:46:04.6213538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T21:46:04.6214643Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T21:46:04.6327556Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T21:46:04.6328638Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T21:46:04.6329809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T21:46:04.6331196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T21:46:04.6332413Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T21:46:04.6333575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T21:46:04.6334759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T21:46:04.6335953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T21:46:04.6337149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T21:46:04.6338538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T21:46:04.6339721Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T21:46:04.6350766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T21:46:04.6426539Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T21:46:04.6427863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:04.6428994Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:04.6430012Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T21:46:04.6430956Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T21:46:04.6431893Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:04.6432779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T21:46:04.6433569Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T21:46:04.6434395Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T21:46:04.6540385Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T21:46:04.6599088Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T21:46:04.6600072Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T21:46:04.6600786Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T21:46:04.6601638Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:04.6602631Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:04.6603585Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:04.6604550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:04.6605563Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:04.6606566Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:04.6607715Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:04.6608849Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:04.6609872Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:04.6610900Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:04.6612253Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:04.6613467Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:04.6614642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:04.6615826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:04.6617050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:04.6618263Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:04.6619255Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:04.6620199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:04.6621176Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T21:46:04.6622452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T21:46:04.6623705Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T21:46:04.6624898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T21:46:04.6626108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T21:46:04.6627283Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T21:46:04.6628458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T21:46:04.6629611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T21:46:04.6630771Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T21:46:04.6631914Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T21:46:04.6637497Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T21:46:04.6696636Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T21:46:04.6697913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:04.6699028Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:04.6700036Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T21:46:04.6700975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T21:46:04.6701896Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:04.6702777Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T21:46:04.6703562Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T21:46:04.6704382Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T21:46:04.6748894Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T21:46:04.6749712Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T21:46:04.6750482Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T21:46:04.6751403Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:46:04.6752486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:46:04.6753516Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T21:46:04.6754559Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:46:04.6755662Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:46:04.6756737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:46:04.6757965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:46:04.6759194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:46:04.6760290Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:46:04.6761389Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:46:04.6762673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:46:04.6764151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:46:04.6765418Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:46:04.6766673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:46:04.6768066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:46:04.6769373Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:46:04.6770473Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:46:04.6771600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T21:46:04.6772658Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T21:46:04.6773861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T21:46:04.6775199Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T21:46:04.6776479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T21:46:04.6777676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T21:46:04.6778922Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T21:46:04.6780178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T21:46:04.6781434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T21:46:04.6782682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T21:46:04.6783907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T21:46:04.6785232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T21:46:04.6889278Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T21:46:04.6890555Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T21:46:04.6891949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T21:46:04.6893343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T21:46:04.6894682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T21:46:04.6895956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T21:46:04.6897262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T21:46:04.6898589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T21:46:04.6899893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T21:46:04.6901206Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T21:46:04.6902509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T21:46:04.6910988Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T21:46:04.6963544Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T21:46:04.6964951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:04.6966170Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:46:04.6967428Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T21:46:04.6968461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T21:46:04.6969467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T21:46:04.6970411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T21:46:04.6971394Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T21:46:04.6972302Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T21:46:04.7065777Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T21:46:04.7106688Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T21:46:04.7107356Z creating: build/lib/ 2024-08-20T21:46:04.7107712Z inflating: build/lib/libclog.a 2024-08-20T21:46:04.7115198Z inflating: build/lib/libpthreadpool.a 2024-08-20T21:46:04.7122845Z inflating: build/lib/libcpuinfo.a 2024-08-20T21:46:04.7129981Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T21:46:04.7194763Z inflating: build/lib/libgtest.a 2024-08-20T21:46:04.7200674Z inflating: build/lib/libittnotify.a 2024-08-20T21:46:04.7279588Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T21:46:04.7346869Z inflating: build/lib/libbenchmark.a 2024-08-20T21:46:04.7373038Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T21:46:04.7433919Z inflating: build/lib/libasmjit.a 2024-08-20T21:46:04.7538666Z inflating: build/lib/libgloo.a 2024-08-20T21:46:04.7954576Z inflating: build/lib/libprotobuf.a 2024-08-20T21:46:04.7974061Z inflating: build/lib/libfmt.a 2024-08-20T21:46:04.8065677Z inflating: build/lib/libc10.so 2024-08-20T21:46:04.8066185Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T21:46:04.8067592Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T21:46:04.8087441Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T21:46:04.8553332Z inflating: build/lib/libprotoc.a 2024-08-20T21:46:04.8570916Z inflating: build/lib/libgmock.a 2024-08-20T21:46:04.8571538Z inflating: build/lib/libgtest_main.a 2024-08-20T21:46:04.8572040Z inflating: build/lib/libbenchmark_main.a 2024-08-20T21:46:04.8588823Z inflating: build/lib/libnnpack.a 2024-08-20T21:46:05.7554114Z inflating: build/lib/libdnnl.a 2024-08-20T21:46:05.8066755Z inflating: build/lib/libtensorpipe.a 2024-08-20T21:46:05.9027439Z inflating: build/lib/libfbgemm.a 2024-08-20T21:46:05.9294630Z inflating: build/lib/libkineto.a 2024-08-20T21:46:05.9295471Z inflating: build/lib/libgmock_main.a 2024-08-20T21:46:05.9333028Z inflating: build/lib/libonnx_proto.a 2024-08-20T21:46:05.9505946Z inflating: build/lib/libXNNPACK.a 2024-08-20T21:46:06.0146063Z inflating: build/lib/libonnx.a 2024-08-20T21:46:08.2049987Z inflating: build/lib/libtorch_cpu.so 2024-08-20T21:46:08.2050887Z inflating: build/lib/libtorch.so 2024-08-20T21:46:08.2055100Z inflating: build/lib/libunbox_lib.a 2024-08-20T21:46:08.2058598Z inflating: build/lib/libshm.so 2024-08-20T21:46:08.2078524Z inflating: build/lib/libjitbackend_test.so 2024-08-20T21:46:08.2143243Z inflating: build/lib/libtorchbind_test.so 2024-08-20T21:46:08.2168340Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T21:46:08.2191801Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T21:46:08.3969026Z inflating: build/lib/libtorch_python.so 2024-08-20T21:46:08.4001789Z inflating: build/lib/libnnapi_backend.so 2024-08-20T21:46:08.4002671Z creating: build/bin/ 2024-08-20T21:46:08.4003086Z creating: build/bin/CMakeFiles/ 2024-08-20T21:46:08.4003788Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:46:08.4004466Z extracting: build/bin/CMakeFiles/progress.marks 2024-08-20T21:46:08.4005172Z inflating: build/bin/Makefile 2024-08-20T21:46:08.4005633Z inflating: build/bin/cmake_install.cmake 2024-08-20T21:46:08.4006132Z inflating: build/bin/CTestTestfile.cmake 2024-08-20T21:46:08.4053354Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T21:46:08.4105067Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T21:46:08.4154750Z inflating: build/bin/c10_Device_test 2024-08-20T21:46:08.4209674Z inflating: build/bin/c10_Scalar_test 2024-08-20T21:46:08.4268446Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T21:46:08.4317166Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T21:46:08.4368529Z inflating: build/bin/c10_SymInt_test 2024-08-20T21:46:08.4420936Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T21:46:08.4476961Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T21:46:08.4532571Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T21:46:08.4579650Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T21:46:08.4633718Z inflating: build/bin/c10_Bitset_test 2024-08-20T21:46:08.4702853Z inflating: build/bin/c10_cow_test 2024-08-20T21:46:08.4751868Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T21:46:08.4802217Z inflating: build/bin/c10_Half_test 2024-08-20T21:46:08.4856592Z inflating: build/bin/c10_LeftRight_test 2024-08-20T21:46:08.4911022Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T21:46:08.4961204Z inflating: build/bin/c10_Synchronized_test 2024-08-20T21:46:08.5015755Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T21:46:08.5068749Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T21:46:08.5118855Z inflating: build/bin/c10_TypeList_test 2024-08-20T21:46:08.5170279Z inflating: build/bin/c10_accumulate_test 2024-08-20T21:46:08.5218168Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T21:46:08.5274243Z inflating: build/bin/c10_bfloat16_test 2024-08-20T21:46:08.5323947Z inflating: build/bin/c10_bit_cast_test 2024-08-20T21:46:08.5379005Z inflating: build/bin/c10_complex_math_test 2024-08-20T21:46:08.5434875Z inflating: build/bin/c10_complex_test 2024-08-20T21:46:08.5487978Z inflating: build/bin/c10_exception_test 2024-08-20T21:46:08.5535498Z inflating: build/bin/c10_flags_test 2024-08-20T21:46:08.5586527Z inflating: build/bin/c10_generic_math_test 2024-08-20T21:46:08.5638579Z inflating: build/bin/c10_lazy_test 2024-08-20T21:46:08.5689113Z inflating: build/bin/c10_irange_test 2024-08-20T21:46:08.5846697Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T21:46:08.5902146Z inflating: build/bin/c10_logging_test 2024-08-20T21:46:08.5976478Z inflating: build/bin/c10_optional_test 2024-08-20T21:46:08.6030325Z inflating: build/bin/c10_registry_test 2024-08-20T21:46:08.6090371Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T21:46:08.6234708Z inflating: build/bin/c10_small_vector_test 2024-08-20T21:46:08.6285675Z inflating: build/bin/c10_ssize_test 2024-08-20T21:46:08.6335652Z inflating: build/bin/c10_string_util_test 2024-08-20T21:46:08.6393886Z inflating: build/bin/c10_string_view_test 2024-08-20T21:46:08.6444379Z inflating: build/bin/c10_tempfile_test 2024-08-20T21:46:08.6490722Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T21:46:08.6546111Z inflating: build/bin/c10_typeid_test 2024-08-20T21:46:08.6975801Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T21:46:08.7405820Z inflating: build/bin/protoc 2024-08-20T21:46:08.7738512Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T21:46:08.8054390Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T21:46:08.8401342Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T21:46:08.8452308Z inflating: build/bin/BackoffTest 2024-08-20T21:46:08.8506178Z inflating: build/bin/FileStoreTest 2024-08-20T21:46:08.8562961Z inflating: build/bin/TCPStoreTest 2024-08-20T21:46:08.8614518Z inflating: build/bin/HashStoreTest 2024-08-20T21:46:08.8616479Z inflating: build/bin/example_allreduce 2024-08-20T21:46:08.8682141Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T21:46:08.8733046Z inflating: build/bin/test_dist_autograd 2024-08-20T21:46:08.8735008Z inflating: build/bin/parallel_benchmark 2024-08-20T21:46:08.8803940Z inflating: build/bin/test_cpp_rpc 2024-08-20T21:46:08.8811134Z inflating: build/bin/aot_model_compiler_test 2024-08-20T21:46:08.8877269Z inflating: build/bin/test_mobile_nnc 2024-08-20T21:46:08.9199572Z inflating: build/bin/test_lazy 2024-08-20T21:46:08.9256522Z inflating: build/bin/static_runtime_bench 2024-08-20T21:46:08.9329360Z inflating: build/bin/Dict_test 2024-08-20T21:46:08.9380722Z inflating: build/bin/Dimname_test 2024-08-20T21:46:08.9619515Z inflating: build/bin/static_runtime_test 2024-08-20T21:46:08.9676905Z inflating: build/bin/NamedTensor_test 2024-08-20T21:46:08.9740338Z inflating: build/bin/MaybeOwned_test 2024-08-20T21:46:08.9799752Z inflating: build/bin/apply_utils_test 2024-08-20T21:46:08.9855826Z inflating: build/bin/atest 2024-08-20T21:46:08.9918628Z inflating: build/bin/basic 2024-08-20T21:46:08.9972531Z inflating: build/bin/broadcast_test 2024-08-20T21:46:09.0022792Z inflating: build/bin/cpu_allocator_test 2024-08-20T21:46:09.0081855Z inflating: build/bin/cpu_generator_test 2024-08-20T21:46:09.0132341Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T21:46:09.0220172Z inflating: build/bin/cpu_rng_test 2024-08-20T21:46:09.0271427Z inflating: build/bin/dispatch_key_set_test 2024-08-20T21:46:09.0320847Z inflating: build/bin/dlconvertor_test 2024-08-20T21:46:09.0376787Z inflating: build/bin/extension_backend_test 2024-08-20T21:46:09.0431804Z inflating: build/bin/half_test 2024-08-20T21:46:09.1505739Z inflating: build/bin/test_api 2024-08-20T21:46:09.1554243Z inflating: build/bin/lazy_tensor_test 2024-08-20T21:46:09.1608969Z inflating: build/bin/math_kernel_test 2024-08-20T21:46:09.1700159Z inflating: build/bin/ivalue_test 2024-08-20T21:46:09.1753550Z inflating: build/bin/memory_format_test 2024-08-20T21:46:09.1805069Z inflating: build/bin/operator_name_test 2024-08-20T21:46:09.1856720Z inflating: build/bin/memory_overlapping_test 2024-08-20T21:46:09.1912817Z inflating: build/bin/native_test 2024-08-20T21:46:09.1966295Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T21:46:09.2015805Z inflating: build/bin/operators_test 2024-08-20T21:46:09.2068358Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T21:46:09.2132740Z inflating: build/bin/pow_test 2024-08-20T21:46:09.2181123Z inflating: build/bin/reduce_ops_test 2024-08-20T21:46:09.2237927Z inflating: build/bin/quantized_test 2024-08-20T21:46:09.2288918Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T21:46:09.2342623Z inflating: build/bin/scalar_tensor_test 2024-08-20T21:46:09.2394506Z inflating: build/bin/StorageUtils_test 2024-08-20T21:46:09.2451366Z inflating: build/bin/scalar_test 2024-08-20T21:46:09.2501806Z inflating: build/bin/stride_properties_test 2024-08-20T21:46:09.2578825Z inflating: build/bin/tensor_iterator_test 2024-08-20T21:46:09.2635699Z inflating: build/bin/type_ptr_test 2024-08-20T21:46:09.2689357Z inflating: build/bin/test_parallel 2024-08-20T21:46:09.2690385Z inflating: build/bin/thread_init_test 2024-08-20T21:46:09.2749914Z inflating: build/bin/type_test 2024-08-20T21:46:09.2803467Z inflating: build/bin/undefined_tensor_test 2024-08-20T21:46:09.2804239Z inflating: build/bin/verify_api_visibility 2024-08-20T21:46:09.2873116Z inflating: build/bin/legacy_vmap_test 2024-08-20T21:46:09.2923991Z inflating: build/bin/weakref_test 2024-08-20T21:46:09.2974442Z inflating: build/bin/wrapdim_test 2024-08-20T21:46:09.3033150Z inflating: build/bin/IListRef_test 2024-08-20T21:46:09.3084396Z inflating: build/bin/xla_tensor_test 2024-08-20T21:46:09.3186886Z inflating: build/bin/List_test 2024-08-20T21:46:09.3301863Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T21:46:09.3394592Z inflating: build/bin/kernel_function_test 2024-08-20T21:46:09.3518115Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T21:46:09.3582695Z inflating: build/bin/KernelFunction_test 2024-08-20T21:46:09.3684228Z inflating: build/bin/kernel_lambda_test 2024-08-20T21:46:09.3733198Z inflating: build/bin/CppSignature_test 2024-08-20T21:46:09.3794271Z inflating: build/bin/kernel_stackbased_test 2024-08-20T21:46:09.3887745Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T21:46:09.3934053Z inflating: build/bin/op_allowlist_test 2024-08-20T21:46:09.3996917Z inflating: build/bin/inline_container_test 2024-08-20T21:46:09.4292380Z inflating: build/bin/op_registration_test 2024-08-20T21:46:09.4347626Z inflating: build/bin/backend_fallback_test 2024-08-20T21:46:09.4402520Z inflating: build/bin/test_edge_op_registration 2024-08-20T21:46:09.4413101Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T21:46:09.4417036Z inflating: build/bin/torch_shm_manager 2024-08-20T21:46:09.5170525Z inflating: build/bin/test_tensorexpr 2024-08-20T21:46:09.5719284Z inflating: build/bin/test_jit 2024-08-20T21:46:09.5719751Z creating: .additional_ci_files/ 2024-08-20T21:46:09.5775404Z inflating: .additional_ci_files/test-times.json 2024-08-20T21:46:09.6003029Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T21:46:09.6072888Z ##[group]Run rm artifacts.zip 2024-08-20T21:46:09.6073268Z rm artifacts.zip 2024-08-20T21:46:09.6079587Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:09.6080075Z env: 2024-08-20T21:46:09.6080333Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:09.6080650Z ##[endgroup] 2024-08-20T21:46:09.6412881Z ##[group]Run df -H 2024-08-20T21:46:09.6413180Z df -H 2024-08-20T21:46:09.6418655Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:09.6419113Z env: 2024-08-20T21:46:09.6419360Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:09.6419669Z ##[endgroup] 2024-08-20T21:46:09.6595816Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T21:46:09.6596546Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T21:46:09.6597074Z tmpfs 50G 0 50G 0% /dev/shm 2024-08-20T21:46:09.6597558Z tmpfs 20G 652k 20G 1% /run 2024-08-20T21:46:09.6598153Z /dev/nvme0n1p1 215G 39G 177G 18% / 2024-08-20T21:46:09.6598588Z tmpfs 50G 21k 50G 1% /tmp 2024-08-20T21:46:09.6599059Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T21:46:09.6640179Z Prepare all required actions 2024-08-20T21:46:09.6640616Z Getting action download info 2024-08-20T21:46:09.7801284Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T21:46:09.7801707Z with: 2024-08-20T21:46:09.7801936Z env: 2024-08-20T21:46:09.7802185Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:09.7802500Z ##[endgroup] 2024-08-20T21:46:09.7893396Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:46:09.7893839Z with: 2024-08-20T21:46:09.7894078Z name: td_results 2024-08-20T21:46:09.7894387Z s3-bucket: gha-artifacts 2024-08-20T21:46:09.7894733Z region: us-east-1 2024-08-20T21:46:09.7895015Z env: 2024-08-20T21:46:09.7895256Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:09.7895588Z ##[endgroup] 2024-08-20T21:46:10.2530755Z (node:485140) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:46:10.2532195Z 2024-08-20T21:46:10.2532543Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:46:10.2533280Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:46:10.2534182Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:46:10.3114065Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T21:46:10.3120186Z Artifact download has finished successfully 2024-08-20T21:46:10.3325431Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T21:46:10.3325864Z mkdir -p .additional_ci_files 2024-08-20T21:46:10.3326376Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T21:46:10.3332426Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:10.3332905Z env: 2024-08-20T21:46:10.3333166Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:10.3333493Z ##[endgroup] 2024-08-20T21:46:10.3568598Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T21:46:10.3581615Z ##[error]Process completed with exit code 1. 2024-08-20T21:46:10.3643896Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T21:46:10.3644353Z .github/scripts/parse_ref.py 2024-08-20T21:46:10.3650144Z shell: /usr/bin/bash -e {0} 2024-08-20T21:46:10.3650470Z env: 2024-08-20T21:46:10.3650730Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:10.3651164Z ##[endgroup] 2024-08-20T21:46:10.3938569Z Prepare all required actions 2024-08-20T21:46:10.4021452Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T21:46:10.4021873Z with: 2024-08-20T21:46:10.4022375Z github-token: *** 2024-08-20T21:46:10.4022644Z env: 2024-08-20T21:46:10.4022874Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:10.4023182Z ##[endgroup] 2024-08-20T21:46:10.4129086Z ##[group]Run set -eux 2024-08-20T21:46:10.4129395Z set -eux 2024-08-20T21:46:10.4129963Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T21:46:10.4135731Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:10.4136206Z env: 2024-08-20T21:46:10.4136464Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:10.4136969Z GITHUB_TOKEN: *** 2024-08-20T21:46:10.4137262Z ##[endgroup] 2024-08-20T21:46:10.4159298Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0315eba2eab835699 2024-08-20T21:46:11.6078917Z setting job-id=29025551055 2024-08-20T21:46:11.6080069Z setting job-name=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:11.6323089Z Prepare all required actions 2024-08-20T21:46:11.6323546Z Getting action download info 2024-08-20T21:46:11.7491816Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T21:46:11.7492310Z with: 2024-08-20T21:46:11.7492782Z github-token: *** 2024-08-20T21:46:11.7513687Z test-matrix: {"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]} 2024-08-20T21:46:11.7534307Z job-name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:11.7535197Z env: 2024-08-20T21:46:11.7535465Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.7535808Z ##[endgroup] 2024-08-20T21:46:11.7768950Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:46:11.7769493Z with: 2024-08-20T21:46:11.7769747Z shell: bash 2024-08-20T21:46:11.7770039Z timeout_minutes: 10 2024-08-20T21:46:11.7770514Z max_attempts: 5 2024-08-20T21:46:11.7770801Z retry_wait_seconds: 30 2024-08-20T21:46:11.7772253Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:46:11.7773480Z polling_interval_seconds: 1 2024-08-20T21:46:11.7773850Z warning_on_retry: true 2024-08-20T21:46:11.7774176Z continue_on_error: false 2024-08-20T21:46:11.7774504Z env: 2024-08-20T21:46:11.7774765Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:11.7775282Z GITHUB_TOKEN: *** 2024-08-20T21:46:11.7775583Z ##[endgroup] 2024-08-20T21:46:11.8480774Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:46:12.0831411Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:46:12.0990862Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T21:46:12.0992679Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T21:46:12.1103386Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T21:46:12.1113063Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T21:46:12.1121498Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T21:46:12.1127274Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T21:46:12.8298887Z Command completed after 1 attempt(s). 2024-08-20T21:46:12.8389488Z ##[group]Run set -x 2024-08-20T21:46:12.8389986Z set -x 2024-08-20T21:46:12.8390278Z  2024-08-20T21:46:12.8390827Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:46:12.8391510Z # in runner workspace 2024-08-20T21:46:12.8392023Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T21:46:12.8399408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:12.8399911Z env: 2024-08-20T21:46:12.8400167Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.8400505Z ##[endgroup] 2024-08-20T21:46:12.8426756Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T21:46:12.8752967Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:46:12.8753579Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:46:12.8754034Z echo "Job name: ${JOB_NAME}" 2024-08-20T21:46:12.8754430Z  2024-08-20T21:46:12.8754976Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:46:12.8755668Z # in runner workspace 2024-08-20T21:46:12.8756245Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T21:46:12.8756896Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T21:46:12.8757349Z  --job-name "${JOB_NAME}" \ 2024-08-20T21:46:12.8777422Z  --test-matrix "{"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]}" \ 2024-08-20T21:46:12.8798092Z  --selected-test-configs "" \ 2024-08-20T21:46:12.8798551Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T21:46:12.8799025Z  --tag "${TAG}" \ 2024-08-20T21:46:12.8799395Z  --event-name "${EVENT_NAME}" \ 2024-08-20T21:46:12.8799836Z  --schedule "${SCHEDULE}" \ 2024-08-20T21:46:12.8800254Z  --branch "${HEAD_BRANCH}" 2024-08-20T21:46:12.8806173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:12.8806626Z env: 2024-08-20T21:46:12.8806883Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:12.8807393Z GITHUB_TOKEN: *** 2024-08-20T21:46:12.8808163Z JOB_NAME: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:12.8808989Z PR_NUMBER: 2024-08-20T21:46:12.8809257Z TAG: ciflow/inductor/133712 2024-08-20T21:46:12.8809595Z EVENT_NAME: push 2024-08-20T21:46:12.8809874Z SCHEDULE: 2024-08-20T21:46:12.8810118Z HEAD_BRANCH: 2024-08-20T21:46:12.8810388Z ##[endgroup] 2024-08-20T21:46:12.8834763Z Workflow: inductor 2024-08-20T21:46:12.8835872Z Job name: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:13.1569246Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T21:46:13.3110890Z ##[group]Run echo "Filtered matrix:" 2024-08-20T21:46:13.3111340Z echo "Filtered matrix:" 2024-08-20T21:46:13.3130300Z echo "{"include": [{"config": "inductor_avx512", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_avx512", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_inductor_huggingface_amp_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_timm_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "cpu_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.16xlarge.spr"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_huggingface_freezing", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_timm_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "dynamic_cpu_aot_inductor_torchbench_amp_freezing", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.12xlarge"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.24xl.spr-metal"}, {"config": "inductor_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "inductor_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_huggingface_freezing_avx2", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_torchbench_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}, {"config": "cpu_inductor_timm_freezing_avx2", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.10xlarge.avx2"}]}" 2024-08-20T21:46:13.3151084Z  2024-08-20T21:46:13.3151332Z echo 2024-08-20T21:46:13.3151711Z echo "Is the current job unstable? False" 2024-08-20T21:46:13.3152273Z  2024-08-20T21:46:13.3152509Z echo 2024-08-20T21:46:13.3152851Z echo "Is keep-going label set? False" 2024-08-20T21:46:13.3153275Z  2024-08-20T21:46:13.3153522Z echo 2024-08-20T21:46:13.3153905Z echo "Renabled issues? " 2024-08-20T21:46:13.3159372Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:13.3159840Z env: 2024-08-20T21:46:13.3160092Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:13.3160521Z ##[endgroup] 2024-08-20T21:46:13.3183116Z Filtered matrix: 2024-08-20T21:46:13.3210360Z {include: [{config: inductor_avx512, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: inductor_avx512, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_timm_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_inductor_huggingface_amp_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_timm_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_timm_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: cpu_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.16xlarge.spr}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_huggingface_freezing, shard: 1, num_shards: 1, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_timm_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_timm_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: cpu_aot_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_amp_freezing, shard: 1, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: dynamic_cpu_aot_inductor_torchbench_amp_freezing, shard: 2, num_shards: 2, runner: amz2023.linux.12xlarge}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: amz2023.linux.24xl.spr-metal}, {config: inductor_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: inductor_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_huggingface_freezing_avx2, shard: 1, num_shards: 1, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_torchbench_freezing_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_torchbench_freezing_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_timm_freezing_avx2, shard: 1, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}, {config: cpu_inductor_timm_freezing_avx2, shard: 2, num_shards: 2, runner: amz2023.linux.10xlarge.avx2}]} 2024-08-20T21:46:13.3231280Z 2024-08-20T21:46:13.3231433Z Is the current job unstable? False 2024-08-20T21:46:13.3231714Z 2024-08-20T21:46:13.3231925Z Is keep-going label set? False 2024-08-20T21:46:13.3254761Z 2024-08-20T21:46:13.3254951Z Renabled issues? 2024-08-20T21:46:13.3322355Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:46:13.3323016Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:46:13.3328721Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:46:13.3329209Z env: 2024-08-20T21:46:13.3329469Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:13.3329806Z JOB_TIMEOUT: 240 2024-08-20T21:46:13.3330077Z ##[endgroup] 2024-08-20T21:46:13.3580197Z ##[group]Run set -x 2024-08-20T21:46:13.3580585Z set -x 2024-08-20T21:46:13.3580863Z  2024-08-20T21:46:13.3581191Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T21:46:13.3581880Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T21:46:13.3582438Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T21:46:13.3583042Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T21:46:13.3583439Z else 2024-08-20T21:46:13.3583743Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:46:13.3584279Z fi 2024-08-20T21:46:13.3584530Z  2024-08-20T21:46:13.3584979Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T21:46:13.3585706Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T21:46:13.3586368Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T21:46:13.3586952Z # shellcheck disable=SC2086,SC2090 2024-08-20T21:46:13.3587381Z container_name=$(docker run \ 2024-08-20T21:46:13.3587876Z  ${GPU_FLAG:-} \ 2024-08-20T21:46:13.3588212Z  -e BUILD_ENVIRONMENT \ 2024-08-20T21:46:13.3588573Z  -e PR_NUMBER \ 2024-08-20T21:46:13.3588884Z  -e GITHUB_ACTIONS \ 2024-08-20T21:46:13.3589235Z  -e GITHUB_REPOSITORY \ 2024-08-20T21:46:13.3589603Z  -e GITHUB_WORKFLOW \ 2024-08-20T21:46:13.3590118Z  -e GITHUB_JOB \ 2024-08-20T21:46:13.3590447Z  -e GITHUB_RUN_ID \ 2024-08-20T21:46:13.3590816Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T21:46:13.3591199Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T21:46:13.3591577Z  -e JOB_ID \ 2024-08-20T21:46:13.3591873Z  -e JOB_NAME \ 2024-08-20T21:46:13.3592183Z  -e BASE_SHA \ 2024-08-20T21:46:13.3592490Z  -e BRANCH \ 2024-08-20T21:46:13.3592790Z  -e SHA1 \ 2024-08-20T21:46:13.3593111Z  -e AWS_DEFAULT_REGION \ 2024-08-20T21:46:13.3593495Z  -e IN_WHEEL_TEST \ 2024-08-20T21:46:13.3593853Z  -e SHARD_NUMBER \ 2024-08-20T21:46:13.3594188Z  -e TEST_CONFIG \ 2024-08-20T21:46:13.3594545Z  -e NUM_TEST_SHARDS \ 2024-08-20T21:46:13.3594921Z  -e REENABLED_ISSUES \ 2024-08-20T21:46:13.3595316Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T21:46:13.3595817Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T21:46:13.3596182Z  -e TEST_SHOWLOCALS \ 2024-08-20T21:46:13.3596538Z  -e NO_TEST_TIMEOUT \ 2024-08-20T21:46:13.3596875Z  -e NO_TD \ 2024-08-20T21:46:13.3597167Z  -e TD_DISTRIBUTED \ 2024-08-20T21:46:13.3597514Z  -e PR_LABELS \ 2024-08-20T21:46:13.3597884Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T21:46:13.3598286Z  -e SCCACHE_BUCKET \ 2024-08-20T21:46:13.3598641Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T21:46:13.3599015Z  -e XLA_CUDA \ 2024-08-20T21:46:13.3599384Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T21:46:13.3599845Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T21:46:13.3600319Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T21:46:13.3600793Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T21:46:13.3601231Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T21:46:13.3601639Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T21:46:13.3602053Z  -e DASHBOARD_TAG \ 2024-08-20T21:46:13.3602505Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T21:46:13.3603025Z  --security-opt seccomp=unconfined \ 2024-08-20T21:46:13.3603446Z  --cap-add=SYS_PTRACE \ 2024-08-20T21:46:13.3603809Z  --ipc=host \ 2024-08-20T21:46:13.3604137Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T21:46:13.3604500Z  --tty \ 2024-08-20T21:46:13.3604772Z  --detach \ 2024-08-20T21:46:13.3605226Z  --name="${container_name}" \ 2024-08-20T21:46:13.3605618Z  --user jenkins \ 2024-08-20T21:46:13.3606071Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T21:46:13.3606582Z  -w /var/lib/jenkins/workspace \ 2024-08-20T21:46:13.3606988Z  "${DOCKER_IMAGE}" 2024-08-20T21:46:13.3607306Z ) 2024-08-20T21:46:13.3607902Z # Propagate download.pytorch.org IP to container 2024-08-20T21:46:13.3608813Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T21:46:13.3610047Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T21:46:13.3611089Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T21:46:13.3616837Z shell: /usr/bin/bash -e {0} 2024-08-20T21:46:13.3617159Z env: 2024-08-20T21:46:13.3617544Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:46:13.3617972Z BUILD_ENVIRONMENT: linux-jammy-py3.8-gcc11-build 2024-08-20T21:46:13.3618428Z PR_NUMBER: 2024-08-20T21:46:13.3618716Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T21:46:13.3619109Z GITHUB_WORKFLOW: inductor 2024-08-20T21:46:13.3619444Z GITHUB_JOB: test 2024-08-20T21:46:13.3619741Z GITHUB_RUN_ID: 10479310725 2024-08-20T21:46:13.3620067Z GITHUB_RUN_NUMBER: 88082 2024-08-20T21:46:13.3620400Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T21:46:13.3620712Z JOB_ID: 29025551055 2024-08-20T21:46:13.3621498Z JOB_NAME: linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:46:13.3622344Z BRANCH: 2024-08-20T21:46:13.3622663Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:46:13.3623171Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:46:13.3623703Z TEST_CONFIG: cpu_aot_inductor_torchbench_amp_freezing 2024-08-20T21:46:13.3624152Z SHARD_NUMBER: 1 2024-08-20T21:46:13.3624439Z NUM_TEST_SHARDS: 2 2024-08-20T21:46:13.3624749Z REENABLED_ISSUES: 2024-08-20T21:46:13.3625065Z CONTINUE_THROUGH_ERROR: False 2024-08-20T21:46:13.3625420Z VERBOSE_TEST_LOGS: False 2024-08-20T21:46:13.3625754Z TEST_SHOWLOCALS: False 2024-08-20T21:46:13.3626080Z NO_TEST_TIMEOUT: False 2024-08-20T21:46:13.3626377Z NO_TD: False 2024-08-20T21:46:13.3626653Z TD_DISTRIBUTED: False 2024-08-20T21:46:13.3627049Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T21:46:13.3627530Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T21:46:13.3627878Z SHM_SIZE: 1g 2024-08-20T21:46:13.3628885Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:46:13.3629985Z XLA_CUDA: 2024-08-20T21:46:13.3630451Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:46:13.3631038Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T21:46:13.3631453Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T21:46:13.3631838Z DASHBOARD_TAG: 2024-08-20T21:46:13.3632330Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T21:46:13.3632694Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T21:46:13.3633033Z ##[endgroup] 2024-08-20T21:46:13.3659624Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \m\u\l\t\i\g\p\u ]] 2024-08-20T21:46:13.3660768Z + [[ linux-jammy-py3.8-gcc11-build == *onnx* ]] 2024-08-20T21:46:13.3661284Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:46:13.3669157Z +++ nproc --ignore=2 2024-08-20T21:46:13.3948992Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:47:32.7521397Z + container_name=b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T21:47:32.7525029Z + grep download.pytorch.org /etc/hosts 2024-08-20T21:47:32.7527330Z + docker exec -i b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T21:47:32.8806443Z + echo DOCKER_CONTAINER_ID=b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T21:47:32.8811935Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-20T21:47:32.8814458Z + docker exec -t b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T21:47:33.2552435Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T21:47:33.6806778Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T21:47:33.6808899Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T21:47:33.6811093Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T21:47:33.6812998Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T21:47:33.6815274Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T21:47:33.6820635Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.12.1) 2024-08-20T21:47:33.6838904Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T21:47:33.6849128Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T21:47:33.6871735Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.21.2) 2024-08-20T21:47:33.7236757Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T21:47:34.3743187Z Installing collected packages: torch 2024-08-20T21:47:41.7757738Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T21:47:41.7759248Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T21:47:41.7759992Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T21:47:41.8398150Z + export TERM=vt100 2024-08-20T21:47:41.8398927Z + TERM=vt100 2024-08-20T21:47:41.8400783Z ++ dirname .ci/pytorch/test.sh 2024-08-20T21:47:41.8403630Z + source .ci/pytorch/common.sh 2024-08-20T21:47:41.8406484Z +++ dirname .ci/pytorch/common.sh 2024-08-20T21:47:41.8412630Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T21:47:41.8413956Z +++ declare -f -t trap_add 2024-08-20T21:47:41.8417317Z ++ set -ex 2024-08-20T21:47:41.8418055Z ++ [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:41.8418553Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T21:47:41.8419022Z + [[ linux-jammy-py3.8-gcc11-build != *rocm* ]] 2024-08-20T21:47:41.8421145Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T21:47:41.8430434Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T21:47:41.8430863Z + trap_add cleanup_workspace EXIT 2024-08-20T21:47:41.8431264Z + trap_add_cmd=cleanup_workspace 2024-08-20T21:47:41.8431606Z + shift 2024-08-20T21:47:41.8431986Z + for trap_add_name in "$@" 2024-08-20T21:47:41.8438859Z +++ trap -p EXIT 2024-08-20T21:47:41.8440675Z ++ eval 'extract_trap_cmd ' 2024-08-20T21:47:41.8441040Z +++ extract_trap_cmd 2024-08-20T21:47:41.8441368Z +++ printf '%s\n' '' 2024-08-20T21:47:41.8441748Z ++ printf '%s\n' cleanup_workspace 2024-08-20T21:47:41.8442707Z + trap -- ' 2024-08-20T21:47:41.8443005Z cleanup_workspace' EXIT 2024-08-20T21:47:41.8443467Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T21:47:42.2826304Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T21:47:42.2844396Z + echo 'Environment variables:' 2024-08-20T21:47:42.2844862Z Environment variables: 2024-08-20T21:47:42.2845464Z + env 2024-08-20T21:47:42.2849006Z INSTALLED_DB=yes 2024-08-20T21:47:42.2849591Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:47:42.2850180Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:47:42.2850779Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11-build 2024-08-20T21:47:42.2851246Z HOSTNAME=b46344175dba 2024-08-20T21:47:42.2852142Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.2852990Z GITHUB_ACTION=__self 2024-08-20T21:47:42.2853326Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:47:42.2853724Z GITHUB_RUN_NUMBER=88082 2024-08-20T21:47:42.2854121Z TEST_CONFIG=cpu_aot_inductor_torchbench_amp_freezing 2024-08-20T21:47:42.2854615Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:47:42.2855089Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:47:42.2855512Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:47:42.2855940Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:42.2856369Z GITHUB_REF_TYPE=tag 2024-08-20T21:47:42.2856684Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:47:42.2857105Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.2857536Z XLA_CUDA= 2024-08-20T21:47:42.2858037Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T21:47:42.2863810Z *** 2024-08-20T21:47:42.2864135Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:47:42.2864495Z GITHUB_ACTIONS=true 2024-08-20T21:47:42.2864853Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.2865365Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.2866176Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T21:47:42.2866906Z UCC_HOME=/usr 2024-08-20T21:47:42.2867175Z VERBOSE_TEST_LOGS=False 2024-08-20T21:47:42.2867540Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T21:47:42.2867955Z SHARD_NUMBER=1 2024-08-20T21:47:42.2868248Z GITHUB_REF_PROTECTED=false 2024-08-20T21:47:42.2868572Z HOME=/var/lib/jenkins 2024-08-20T21:47:42.2868935Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:47:42.2869362Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:47:42.2869726Z UCX_COMMIT= 2024-08-20T21:47:42.2870007Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T21:47:42.2870362Z NUM_TEST_SHARDS=2 2024-08-20T21:47:42.2870646Z UCX_HOME=/usr 2024-08-20T21:47:42.2871707Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.2873165Z JOB_NAME=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:47:42.2874592Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.2875759Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:47:42.2876405Z GITHUB_EVENT_NAME=push 2024-08-20T21:47:42.2876718Z DASHBOARD_TAG= 2024-08-20T21:47:42.2877130Z GITHUB_RUN_ID=10479310725 2024-08-20T21:47:42.2878091Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.2879026Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:42.2879372Z PR_NUMBER= 2024-08-20T21:47:42.2879631Z DESIRED_CUDA= 2024-08-20T21:47:42.2879910Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:47:42.2880221Z ANACONDA_PYTHON_VERSION=3.8 2024-08-20T21:47:42.2880657Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:47:42.2881108Z TERM=vt100 2024-08-20T21:47:42.2881374Z INSTALLED_VISION=yes 2024-08-20T21:47:42.2881656Z BRANCH= 2024-08-20T21:47:42.2881922Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:47:42.2882284Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:47:42.2883046Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:47:42.2883809Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:47:42.2884196Z UCC_COMMIT= 2024-08-20T21:47:42.2884467Z REENABLED_ISSUES= 2024-08-20T21:47:42.2884735Z DOCS=yes 2024-08-20T21:47:42.2884993Z INSTALLED_ANDROID= 2024-08-20T21:47:42.2885277Z SHLVL=1 2024-08-20T21:47:42.2885508Z MAX_JOBS=46 2024-08-20T21:47:42.2885778Z GITHUB_ACTOR_ID=54816060 2024-08-20T21:47:42.2886227Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.2886764Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T21:47:42.2887424Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:47:42.2888005Z GITHUB_JOB=test 2024-08-20T21:47:42.2888292Z NO_TEST_TIMEOUT=False 2024-08-20T21:47:42.2888605Z TD_DISTRIBUTED=False 2024-08-20T21:47:42.2888927Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:47:42.2889312Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:47:42.2889645Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:47:42.2889971Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:47:42.2891162Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:42.2892275Z GITHUB_BASE_REF= 2024-08-20T21:47:42.2892560Z INSTALLED_ACL= 2024-08-20T21:47:42.2892818Z CI=true 2024-08-20T21:47:42.2893092Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:47:42.2893457Z JOB_ID=29025551055 2024-08-20T21:47:42.2893752Z INSTALLED_PROTOBUF=yes 2024-08-20T21:47:42.2894054Z GITHUB_HEAD_REF= 2024-08-20T21:47:42.2894338Z GITHUB_ACTION_REF= 2024-08-20T21:47:42.2894789Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:47:42.2895243Z TEST_SHOWLOCALS=False 2024-08-20T21:47:42.2895558Z GITHUB_WORKFLOW=inductor 2024-08-20T21:47:42.2895904Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:47:42.2896848Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.2897669Z NO_TD=False 2024-08-20T21:47:42.2897960Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:47:42.2898310Z _=/usr/bin/env 2024-08-20T21:47:42.2898787Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T21:47:42.3084331Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-08-20T21:47:42.3085272Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-20T21:47:42.3086156Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-08-20T21:47:42.3087059Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-08-20T21:47:42.3087856Z + BUILD_DIR=build 2024-08-20T21:47:42.3088170Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T21:47:42.3088552Z + BUILD_BIN_DIR=build/bin 2024-08-20T21:47:42.3088877Z + SHARD_NUMBER=1 2024-08-20T21:47:42.3089158Z + NUM_TEST_SHARDS=2 2024-08-20T21:47:42.3089460Z + export VALGRIND=ON 2024-08-20T21:47:42.3089762Z + VALGRIND=ON 2024-08-20T21:47:42.3090191Z + [[ linux-jammy-py3.8-gcc11-build == *clang9* ]] 2024-08-20T21:47:42.3090626Z + [[ 0 == \1 ]] 2024-08-20T21:47:42.3091032Z + [[ False == \1 ]] 2024-08-20T21:47:42.3091584Z + [[ linux-jammy-py3.8-gcc11-build != *bazel* ]] 2024-08-20T21:47:42.3092077Z ++ realpath build/custom_test_artifacts 2024-08-20T21:47:42.3093039Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T21:47:42.3093726Z + [[ -n '' ]] 2024-08-20T21:47:42.3094072Z + echo 'Environment variables' 2024-08-20T21:47:42.3094438Z Environment variables 2024-08-20T21:47:42.3094725Z + env 2024-08-20T21:47:42.3099244Z INSTALLED_DB=yes 2024-08-20T21:47:42.3099809Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:47:42.3100417Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:47:42.3100888Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11-build 2024-08-20T21:47:42.3101345Z HOSTNAME=b46344175dba 2024-08-20T21:47:42.3102219Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.3103040Z GITHUB_ACTION=__self 2024-08-20T21:47:42.3103359Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:47:42.3103750Z GITHUB_RUN_NUMBER=88082 2024-08-20T21:47:42.3104160Z TEST_CONFIG=cpu_aot_inductor_torchbench_amp_freezing 2024-08-20T21:47:42.3104651Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:47:42.3105112Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:47:42.3105535Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:47:42.3105979Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:42.3106375Z GITHUB_REF_TYPE=tag 2024-08-20T21:47:42.3106690Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:47:42.3107110Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.3107544Z XLA_CUDA= 2024-08-20T21:47:42.3108001Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T21:47:42.3108381Z *** 2024-08-20T21:47:42.3108645Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:47:42.3108996Z GITHUB_ACTIONS=true 2024-08-20T21:47:42.3109340Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.3109841Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.3110646Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T21:47:42.3111380Z UCC_HOME=/usr 2024-08-20T21:47:42.3111650Z VERBOSE_TEST_LOGS=False 2024-08-20T21:47:42.3112015Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T21:47:42.3112426Z SHARD_NUMBER=1 2024-08-20T21:47:42.3112719Z GITHUB_REF_PROTECTED=false 2024-08-20T21:47:42.3113046Z HOME=/var/lib/jenkins 2024-08-20T21:47:42.3113397Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:47:42.3113832Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:47:42.3114202Z UCX_COMMIT= 2024-08-20T21:47:42.3114486Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T21:47:42.3114835Z NUM_TEST_SHARDS=2 2024-08-20T21:47:42.3115117Z UCX_HOME=/usr 2024-08-20T21:47:42.3115965Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.3117404Z JOB_NAME=linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_aot_inductor_torchbench_amp_freezing, 1, 2, amz2023.linux.12xlarge) 2024-08-20T21:47:42.3118866Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.3120033Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:47:42.3120693Z GITHUB_EVENT_NAME=push 2024-08-20T21:47:42.3121008Z DASHBOARD_TAG= 2024-08-20T21:47:42.3121282Z GITHUB_RUN_ID=10479310725 2024-08-20T21:47:42.3122341Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.3123291Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T21:47:42.3123639Z PR_NUMBER= 2024-08-20T21:47:42.3123887Z DESIRED_CUDA= 2024-08-20T21:47:42.3124164Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:47:42.3124468Z VALGRIND=ON 2024-08-20T21:47:42.3124736Z ANACONDA_PYTHON_VERSION=3.8 2024-08-20T21:47:42.3125168Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:47:42.3125622Z TERM=vt100 2024-08-20T21:47:42.3125885Z INSTALLED_VISION=yes 2024-08-20T21:47:42.3127175Z BRANCH= 2024-08-20T21:47:42.3127446Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:47:42.3127811Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:47:42.3128593Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:47:42.3129360Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:47:42.3129747Z UCC_COMMIT= 2024-08-20T21:47:42.3130010Z REENABLED_ISSUES= 2024-08-20T21:47:42.3130286Z DOCS=yes 2024-08-20T21:47:42.3130546Z INSTALLED_ANDROID= 2024-08-20T21:47:42.3130947Z SHLVL=1 2024-08-20T21:47:42.3131191Z MAX_JOBS=46 2024-08-20T21:47:42.3131448Z GITHUB_ACTOR_ID=54816060 2024-08-20T21:47:42.3131898Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:47:42.3132434Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T21:47:42.3133105Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:47:42.3133681Z GITHUB_JOB=test 2024-08-20T21:47:42.3133969Z NO_TEST_TIMEOUT=False 2024-08-20T21:47:42.3134289Z TD_DISTRIBUTED=False 2024-08-20T21:47:42.3134612Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:47:42.3134998Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:47:42.3135333Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:47:42.3135668Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:47:42.3136711Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:42.3137816Z GITHUB_BASE_REF= 2024-08-20T21:47:42.3138107Z INSTALLED_ACL= 2024-08-20T21:47:42.3138365Z CI=true 2024-08-20T21:47:42.3138638Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:47:42.3139000Z JOB_ID=29025551055 2024-08-20T21:47:42.3139295Z INSTALLED_PROTOBUF=yes 2024-08-20T21:47:42.3139600Z GITHUB_HEAD_REF= 2024-08-20T21:47:42.3139884Z GITHUB_ACTION_REF= 2024-08-20T21:47:42.3140322Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:47:42.3140787Z TEST_SHOWLOCALS=False 2024-08-20T21:47:42.3141094Z GITHUB_WORKFLOW=inductor 2024-08-20T21:47:42.3141444Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:47:42.3142391Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_273730f4-dfa0-46c9-9174-e7bcb69bd9fb 2024-08-20T21:47:42.3143224Z NO_TD=False 2024-08-20T21:47:42.3143500Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:47:42.3143847Z _=/usr/bin/env 2024-08-20T21:47:42.3144174Z + echo 'Testing pytorch' 2024-08-20T21:47:42.3144491Z Testing pytorch 2024-08-20T21:47:42.3144804Z + export LANG=C.UTF-8 2024-08-20T21:47:42.3145129Z + LANG=C.UTF-8 2024-08-20T21:47:42.3145580Z + PR_NUMBER= 2024-08-20T21:47:42.3145984Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \d\e\f\a\u\l\t ]] 2024-08-20T21:47:42.3146683Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T21:47:42.3147359Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \s\l\o\w ]] 2024-08-20T21:47:42.3148035Z + [[ linux-jammy-py3.8-gcc11-build == *slow-gradcheck* ]] 2024-08-20T21:47:42.3148644Z + [[ linux-jammy-py3.8-gcc11-build == *cuda* ]] 2024-08-20T21:47:42.3149215Z + [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:42.3149772Z + [[ linux-jammy-py3.8-gcc11-build == *xpu* ]] 2024-08-20T21:47:42.3150320Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *crossref* ]] 2024-08-20T21:47:42.3150941Z + [[ linux-jammy-py3.8-gcc11-build == *rocm* ]] 2024-08-20T21:47:42.3151498Z + [[ linux-jammy-py3.8-gcc11-build == *xpu* ]] 2024-08-20T21:47:42.3152234Z + [[ linux-jammy-py3.8-gcc11-build != *-bazel-* ]] 2024-08-20T21:47:42.3152763Z + pip_install --user ninja==1.10.2 2024-08-20T21:47:42.3153299Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T21:47:42.7156185Z Collecting ninja==1.10.2 2024-08-20T21:47:42.7289553Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T21:47:42.7402381Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T21:47:43.3653856Z Installing collected packages: ninja 2024-08-20T21:47:43.3733959Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T21:47:43.3735294Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T21:47:43.3792953Z Successfully installed ninja-1.10.2 2024-08-20T21:47:43.4399931Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:43.4402179Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:43.4403741Z + [[ linux-jammy-py3.8-gcc11-build == *aarch64* ]] 2024-08-20T21:47:43.4404195Z + install_tlparse 2024-08-20T21:47:43.4404572Z + pip_install --user tlparse==0.3.25 2024-08-20T21:47:43.4405142Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T21:47:43.9759364Z Collecting tlparse==0.3.25 2024-08-20T21:47:43.9872040Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T21:47:44.0084637Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T21:47:44.6999967Z Installing collected packages: tlparse 2024-08-20T21:47:44.8470304Z Successfully installed tlparse-0.3.25 2024-08-20T21:47:44.9130166Z ++ python -m site --user-base 2024-08-20T21:47:44.9403112Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:47:44.9404810Z + [[ linux-jammy-py3.8-gcc11-build == *asan* ]] 2024-08-20T21:47:44.9405392Z + [[ linux-jammy-py3.8-gcc11-build == *-debug* ]] 2024-08-20T21:47:44.9405974Z + [[ linux-jammy-py3.8-gcc11-build != *-bazel-* ]] 2024-08-20T21:47:44.9406858Z + echo 'We are not in debug mode: linux-jammy-py3.8-gcc11-build. Expect the assertion to pass' 2024-08-20T21:47:44.9407931Z We are not in debug mode: linux-jammy-py3.8-gcc11-build. Expect the assertion to pass 2024-08-20T21:47:44.9408583Z + cd test 2024-08-20T21:47:44.9409105Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T21:47:46.3052620Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T21:47:46.3053505Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T21:47:46.3054105Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T21:47:46.3054638Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *pr_time_benchmarks* ]] 2024-08-20T21:47:46.3055324Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *dynamo_eager* ]] 2024-08-20T21:47:46.3055971Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *aot_eager* ]] 2024-08-20T21:47:46.3056622Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *aot_inductor* ]] 2024-08-20T21:47:46.3057469Z + DYNAMO_BENCHMARK_FLAGS+=(--export-aot-inductor) 2024-08-20T21:47:46.3058027Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *dynamic* ]] 2024-08-20T21:47:46.3058622Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *cpu* ]] 2024-08-20T21:47:46.3059196Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2024-08-20T21:47:46.3078242Z + [[ linux-jammy-py3.8-gcc11-build == *libtorch* ]] 2024-08-20T21:47:46.3079188Z + [[ linux-jammy-py3.8-gcc11-build == *-bazel-* ]] 2024-08-20T21:47:46.3080055Z + cd test 2024-08-20T21:47:46.3080738Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T21:47:47.4651807Z PyTorch built with: 2024-08-20T21:47:47.4652396Z - GCC 11.4 2024-08-20T21:47:47.4652730Z - C++ Version: 201703 2024-08-20T21:47:47.4653639Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:47:47.4654767Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:47:47.4655791Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:47:47.4656311Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T21:47:47.4656795Z - NNPACK is enabled 2024-08-20T21:47:47.4657179Z - CPU capability usage: AVX512 2024-08-20T21:47:47.4664957Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T21:47:47.4671646Z 2024-08-20T21:47:47.6502894Z + cd test 2024-08-20T21:47:47.6503706Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T21:47:48.8002512Z ATen/Parallel: 2024-08-20T21:47:48.8002945Z at::get_num_threads() : 24 2024-08-20T21:47:48.8003434Z at::get_num_interop_threads() : 24 2024-08-20T21:47:48.8003837Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:47:48.8004234Z omp_get_max_threads() : 24 2024-08-20T21:47:48.8005371Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:47:48.8006202Z mkl_get_max_threads() : 24 2024-08-20T21:47:48.8006867Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:47:48.8007513Z std::thread::hardware_concurrency() : 48 2024-08-20T21:47:48.8007935Z Environment variables: 2024-08-20T21:47:48.8008273Z OMP_NUM_THREADS : [not set] 2024-08-20T21:47:48.8008634Z MKL_NUM_THREADS : [not set] 2024-08-20T21:47:48.8009000Z ATen parallel backend: OpenMP 2024-08-20T21:47:48.8009260Z 2024-08-20T21:47:48.9828752Z + [[ linux-jammy-py3.8-gcc11-build == *aarch64* ]] 2024-08-20T21:47:48.9829398Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *backward* ]] 2024-08-20T21:47:48.9830020Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *xla* ]] 2024-08-20T21:47:48.9830626Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *executorch* ]] 2024-08-20T21:47:48.9831304Z + [[ cpu_aot_inductor_torchbench_amp_freezing == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T21:47:48.9832008Z + [[ linux-jammy-py3.8-gcc11-build == *libtorch* ]] 2024-08-20T21:47:48.9832599Z + [[ cpu_aot_inductor_torchbench_amp_freezing == distributed ]] 2024-08-20T21:47:48.9833283Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *inductor_distributed* ]] 2024-08-20T21:47:48.9834070Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *inductor-halide* ]] 2024-08-20T21:47:48.9835149Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *inductor-micro-benchmark* ]] 2024-08-20T21:47:48.9835890Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *huggingface* ]] 2024-08-20T21:47:48.9836509Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *timm* ]] 2024-08-20T21:47:48.9837123Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *torchbench* ]] 2024-08-20T21:47:48.9837721Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *cpu* ]] 2024-08-20T21:47:48.9838208Z + install_torchaudio cpu 2024-08-20T21:47:48.9838540Z + local commit 2024-08-20T21:47:48.9838832Z ++ get_pinned_commit audio 2024-08-20T21:47:48.9839289Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T21:47:48.9841071Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:48.9841526Z + [[ cpu == \c\u\d\a ]] 2024-08-20T21:47:48.9842448Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:48.9843970Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:49.3081861Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:49.3083551Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-b9yllt2a 2024-08-20T21:47:49.3117530Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-b9yllt2a 2024-08-20T21:47:50.0296311Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T21:47:50.0333855Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:50.1194338Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T21:47:50.1195419Z Running command git submodule update --init --recursive -q 2024-08-20T21:47:52.0532780Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:47:52.0574761Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T21:47:52.0607051Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T21:47:52.0608899Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T21:47:52.0610919Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T21:47:52.0613624Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T21:47:52.0617062Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T21:47:52.0623123Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.12.1) 2024-08-20T21:47:52.0639820Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T21:47:52.1000354Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T21:47:52.1059632Z Building wheels for collected packages: torchaudio 2024-08-20T21:48:46.1377025Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T21:48:46.1433188Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp38-cp38-linux_x86_64.whl size=1767364 sha256=3b0d6996e6fdf97710ca201cc6bc464b5845a0f6438d2994ddbe9c270a56e7a7 2024-08-20T21:48:46.1434926Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/2d/7d/0d/bdfdb3f2b184f238c1d61b3f9b248992c036dfdb76e5397d45 2024-08-20T21:48:46.1471248Z Successfully built torchaudio 2024-08-20T21:48:46.7302626Z Installing collected packages: torchaudio 2024-08-20T21:48:46.9233488Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T21:48:47.0369947Z + install_torchtext 2024-08-20T21:48:47.0370295Z + local data_commit 2024-08-20T21:48:47.0370598Z + local text_commit 2024-08-20T21:48:47.0371381Z ++ get_pinned_commit data 2024-08-20T21:48:47.0371765Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T21:48:47.0382649Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.0385016Z ++ get_pinned_commit text 2024-08-20T21:48:47.0385607Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T21:48:47.0394927Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:48:47.0396162Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.0397684Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.3592376Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.3594218Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-tw7ggqln 2024-08-20T21:48:47.3627961Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-tw7ggqln 2024-08-20T21:48:47.7718634Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T21:48:47.7750607Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:47.8577963Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:48:48.0048507Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T21:49:35.4643462Z Running command git submodule update --init --recursive -q 2024-08-20T21:49:35.4644270Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:49:35.4692308Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T21:49:35.4694010Z Requirement already satisfied: requests in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T21:49:35.4696966Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T21:49:35.4732628Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T21:49:35.4734504Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T21:49:35.4736776Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T21:49:35.4739564Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T21:49:35.4742696Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T21:49:35.4749731Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-08-20T21:49:35.4763812Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T21:49:35.4819455Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T21:49:35.4822694Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T21:49:35.4827587Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T21:49:35.5212406Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T21:49:35.5280327Z Building wheels for collected packages: torchdata 2024-08-20T21:49:37.1535068Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T21:49:37.1549111Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=1eed21af86417c15c7d17841bb74fd934dd9e608db7b48ad195b770878904917 2024-08-20T21:49:37.1550769Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/f2/b8/b4/f1929a760cf6ea9bda553072159f9dc096899c37ad8a5ffafe 2024-08-20T21:49:37.1588846Z Successfully built torchdata 2024-08-20T21:49:37.7173827Z Installing collected packages: torchdata 2024-08-20T21:49:37.8083219Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T21:49:39.8373380Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:39.8374966Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:40.1583427Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:40.1587917Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-xhx9wmdb 2024-08-20T21:49:40.1620049Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-xhx9wmdb 2024-08-20T21:49:41.0568263Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T21:49:41.0604849Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:41.1496284Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:41.2847478Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T21:49:41.2848565Z Running command git submodule update --init --recursive -q 2024-08-20T21:49:47.7329700Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:49:47.7376032Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T21:49:47.7379037Z Requirement already satisfied: requests in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T21:49:47.7382508Z Requirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T21:49:47.7385537Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-08-20T21:49:47.7388959Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.8/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T21:49:47.7419133Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T21:49:47.7421184Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T21:49:47.7425087Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T21:49:47.7428979Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T21:49:47.7456469Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T21:49:47.7460273Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T21:49:47.7463344Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T21:49:47.7466594Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T21:49:47.7470301Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T21:49:47.7475835Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-08-20T21:49:47.7491591Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T21:49:47.7962024Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T21:49:47.8148980Z Building wheels for collected packages: torchtext 2024-08-20T21:50:16.1647540Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T21:50:16.1712099Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp38-cp38-linux_x86_64.whl size=2045014 sha256=6e1b5870bda6686442f135ece33a18560d4585eb00c79893a26fd58afb9d925a 2024-08-20T21:50:16.1713817Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d6/1b/67/0243915f173a4bc67968d5441b501e3df9c77a50521995f939 2024-08-20T21:50:16.1744177Z Successfully built torchtext 2024-08-20T21:50:16.7295654Z Installing collected packages: torchtext 2024-08-20T21:50:16.8404625Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T21:50:16.9475692Z + install_torchvision 2024-08-20T21:50:16.9476099Z + local orig_preload 2024-08-20T21:50:16.9476396Z + local commit 2024-08-20T21:50:16.9482430Z ++ get_pinned_commit vision 2024-08-20T21:50:16.9482872Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T21:50:16.9485517Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:16.9485950Z + orig_preload= 2024-08-20T21:50:16.9486404Z + '[' -n '' ']' 2024-08-20T21:50:16.9487291Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:16.9488824Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:17.2703326Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:17.2707861Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-_gkvcaep 2024-08-20T21:50:17.2740042Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-_gkvcaep 2024-08-20T21:50:18.6174438Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T21:50:18.6210949Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:18.7132993Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:19.0137778Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T21:50:21.0371941Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:50:21.0417667Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-08-20T21:50:21.0420692Z Requirement already satisfied: torch in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T21:50:21.0425246Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T21:50:21.0496161Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T21:50:21.0500282Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T21:50:21.0503293Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T21:50:21.0506673Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T21:50:21.0509656Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T21:50:21.0515051Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-08-20T21:50:21.0530930Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T21:50:21.0893973Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T21:50:21.0956680Z Building wheels for collected packages: torchvision 2024-08-20T21:50:48.4407225Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T21:50:48.4448365Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp38-cp38-linux_x86_64.whl size=1215626 sha256=73d117870d7cceb36cedb7b2abea2683744432484943beb3e75356a09a88c86e 2024-08-20T21:50:48.4450096Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/82/f9/1e/f126983e23cfd567a4cb268f7d0778919a2a7014fe9af04f11 2024-08-20T21:50:48.4618484Z Successfully built torchvision 2024-08-20T21:50:49.0257627Z Installing collected packages: torchvision 2024-08-20T21:50:49.4111912Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T21:50:49.5020080Z + '[' -n '' ']' 2024-08-20T21:50:49.5020566Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T21:50:49.5022051Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T21:50:49.5022906Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T21:50:49.8255507Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T21:50:49.8259588Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-nb5zdhy7 2024-08-20T21:50:49.8295139Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-nb5zdhy7 2024-08-20T21:50:50.4135308Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T21:50:52.3431465Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T21:50:52.3481368Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T21:51:07.0335411Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | done 2024-08-20T21:51:07.0440748Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp38-cp38-linux_x86_64.whl size=3735230 sha256=04973f3421a6d98bb457bf0456623d85a0f7eb2bf667975c86842851baa0dc31 2024-08-20T21:51:07.0442829Z Stored in directory: /tmp/pip-ephem-wheel-cache-tuz2_4hn/wheels/7a/d0/ee/d845e81513be33ad662e10ba9fb85a811ff1ad92ea52f8b6a9 2024-08-20T21:51:07.0480432Z Successfully built torchao 2024-08-20T21:51:07.6809077Z Installing collected packages: torchao 2024-08-20T21:51:07.9518390Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T21:51:08.0303197Z + id=0 2024-08-20T21:51:08.0303809Z + pip_install opencv-python==4.8.0.74 2024-08-20T21:51:08.0304439Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T21:51:08.4623887Z Collecting opencv-python==4.8.0.74 2024-08-20T21:51:08.4737248Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T21:51:08.4819465Z Requirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-08-20T21:51:08.4910664Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T21:51:09.6927011Z Installing collected packages: opencv-python 2024-08-20T21:51:10.5489748Z Successfully installed opencv-python-4.8.0.74 2024-08-20T21:51:10.6240183Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T21:51:10.6241198Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T21:51:10.6242088Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *torchbench_gcp_smoketest* ]] 2024-08-20T21:51:10.6242713Z + checkout_install_torchbench 2024-08-20T21:51:10.6243070Z + local commit 2024-08-20T21:51:10.6243384Z ++ get_pinned_commit torchbench 2024-08-20T21:51:10.6243809Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T21:51:10.6250683Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T21:51:10.6251391Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T21:51:10.6268748Z Cloning into 'torchbench'... 2024-08-20T21:51:10.7895066Z remote: Enumerating objects: 30997, done. 2024-08-20T21:51:10.7895650Z remote: Counting objects: 0% (1/1737) 2024-08-20T21:51:10.7896174Z remote: Counting objects: 1% (18/1737) 2024-08-20T21:51:10.7896981Z remote: Counting objects: 2% (35/1737) 2024-08-20T21:51:10.7898886Z remote: Counting objects: 3% (53/1737) 2024-08-20T21:51:10.7899425Z remote: Counting objects: 4% (70/1737) 2024-08-20T21:51:10.7899924Z remote: Counting objects: 5% (87/1737) 2024-08-20T21:51:10.7900438Z remote: Counting objects: 6% (105/1737) 2024-08-20T21:51:10.7901448Z remote: Counting objects: 7% (122/1737) 2024-08-20T21:51:10.7902024Z remote: Counting objects: 8% (139/1737) 2024-08-20T21:51:10.7902559Z remote: Counting objects: 9% (157/1737) 2024-08-20T21:51:10.7903067Z remote: Counting objects: 10% (174/1737) 2024-08-20T21:51:10.7903607Z remote: Counting objects: 11% (192/1737) 2024-08-20T21:51:10.7904145Z remote: Counting objects: 12% (209/1737) 2024-08-20T21:51:10.7904666Z remote: Counting objects: 13% (226/1737) 2024-08-20T21:51:10.7905169Z remote: Counting objects: 14% (244/1737) 2024-08-20T21:51:10.7905682Z remote: Counting objects: 15% (261/1737) 2024-08-20T21:51:10.7906195Z remote: Counting objects: 16% (278/1737) 2024-08-20T21:51:10.7906712Z remote: Counting objects: 17% (296/1737) 2024-08-20T21:51:10.7907478Z remote: Counting objects: 18% (313/1737) 2024-08-20T21:51:10.7908019Z remote: Counting objects: 19% (331/1737) 2024-08-20T21:51:10.7908541Z remote: Counting objects: 20% (348/1737) 2024-08-20T21:51:10.7909059Z remote: Counting objects: 21% (365/1737) 2024-08-20T21:51:10.7909562Z remote: Counting objects: 22% (383/1737) 2024-08-20T21:51:10.7910078Z remote: Counting objects: 23% (400/1737) 2024-08-20T21:51:10.7910590Z remote: Counting objects: 24% (417/1737) 2024-08-20T21:51:10.7911134Z remote: Counting objects: 25% (435/1737) 2024-08-20T21:51:10.7911776Z remote: Counting objects: 26% (452/1737) 2024-08-20T21:51:10.7912279Z remote: Counting objects: 27% (469/1737) 2024-08-20T21:51:10.7912793Z remote: Counting objects: 28% (487/1737) 2024-08-20T21:51:10.7913308Z remote: Counting objects: 29% (504/1737) 2024-08-20T21:51:10.7913824Z remote: Counting objects: 30% (522/1737) 2024-08-20T21:51:10.7914328Z remote: Counting objects: 31% (539/1737) 2024-08-20T21:51:10.7914842Z remote: Counting objects: 32% (556/1737) 2024-08-20T21:51:10.7915354Z remote: Counting objects: 33% (574/1737) 2024-08-20T21:51:10.7915865Z remote: Counting objects: 34% (591/1737) 2024-08-20T21:51:10.7916365Z remote: Counting objects: 35% (608/1737) 2024-08-20T21:51:10.7916879Z remote: Counting objects: 36% (626/1737) 2024-08-20T21:51:10.7917391Z remote: Counting objects: 37% (643/1737) 2024-08-20T21:51:10.7917890Z remote: Counting objects: 38% (661/1737) 2024-08-20T21:51:10.7918402Z remote: Counting objects: 39% (678/1737) 2024-08-20T21:51:10.7918921Z remote: Counting objects: 40% (695/1737) 2024-08-20T21:51:10.7919437Z remote: Counting objects: 41% (713/1737) 2024-08-20T21:51:10.7919932Z remote: Counting objects: 42% (730/1737) 2024-08-20T21:51:10.7920442Z remote: Counting objects: 43% (747/1737) 2024-08-20T21:51:10.7920959Z remote: Counting objects: 44% (765/1737) 2024-08-20T21:51:10.7921475Z remote: Counting objects: 45% (782/1737) 2024-08-20T21:51:10.7921976Z remote: Counting objects: 46% (800/1737) 2024-08-20T21:51:10.7922488Z remote: Counting objects: 47% (817/1737) 2024-08-20T21:51:10.7923000Z remote: Counting objects: 48% (834/1737) 2024-08-20T21:51:10.7923515Z remote: Counting objects: 49% (852/1737) 2024-08-20T21:51:10.7924015Z remote: Counting objects: 50% (869/1737) 2024-08-20T21:51:10.7924525Z remote: Counting objects: 51% (886/1737) 2024-08-20T21:51:10.7925038Z remote: Counting objects: 52% (904/1737) 2024-08-20T21:51:10.7925563Z remote: Counting objects: 53% (921/1737) 2024-08-20T21:51:10.7926071Z remote: Counting objects: 54% (938/1737) 2024-08-20T21:51:10.7926579Z remote: Counting objects: 55% (956/1737) 2024-08-20T21:51:10.7927093Z remote: Counting objects: 56% (973/1737) 2024-08-20T21:51:10.7927603Z remote: Counting objects: 57% (991/1737) 2024-08-20T21:51:10.7928121Z remote: Counting objects: 58% (1008/1737) 2024-08-20T21:51:10.7928640Z remote: Counting objects: 59% (1025/1737) 2024-08-20T21:51:10.7929170Z remote: Counting objects: 60% (1043/1737) 2024-08-20T21:51:10.7929694Z remote: Counting objects: 61% (1060/1737) 2024-08-20T21:51:10.7930210Z remote: Counting objects: 62% (1077/1737) 2024-08-20T21:51:10.7930717Z remote: Counting objects: 63% (1095/1737) 2024-08-20T21:51:10.7931347Z remote: Counting objects: 64% (1112/1737) 2024-08-20T21:51:10.7931881Z remote: Counting objects: 65% (1130/1737) 2024-08-20T21:51:10.7932406Z remote: Counting objects: 66% (1147/1737) 2024-08-20T21:51:10.7932922Z remote: Counting objects: 67% (1164/1737) 2024-08-20T21:51:10.7933443Z remote: Counting objects: 68% (1182/1737) 2024-08-20T21:51:10.7933965Z remote: Counting objects: 69% (1199/1737) 2024-08-20T21:51:10.7934471Z remote: Counting objects: 70% (1216/1737) 2024-08-20T21:51:10.7934992Z remote: Counting objects: 71% (1234/1737) 2024-08-20T21:51:10.7935513Z remote: Counting objects: 72% (1251/1737) 2024-08-20T21:51:10.7936139Z remote: Counting objects: 73% (1269/1737) 2024-08-20T21:51:10.7936652Z remote: Counting objects: 74% (1286/1737) 2024-08-20T21:51:10.7937171Z remote: Counting objects: 75% (1303/1737) 2024-08-20T21:51:10.7937693Z remote: Counting objects: 76% (1321/1737) 2024-08-20T21:51:10.7938214Z remote: Counting objects: 77% (1338/1737) 2024-08-20T21:51:10.7938719Z remote: Counting objects: 78% (1355/1737) 2024-08-20T21:51:10.7939238Z remote: Counting objects: 79% (1373/1737) 2024-08-20T21:51:10.7939761Z remote: Counting objects: 80% (1390/1737) 2024-08-20T21:51:10.7940358Z remote: Counting objects: 81% (1407/1737) 2024-08-20T21:51:10.7940863Z remote: Counting objects: 82% (1425/1737) 2024-08-20T21:51:10.7941384Z remote: Counting objects: 83% (1442/1737) 2024-08-20T21:51:10.7941908Z remote: Counting objects: 84% (1460/1737) 2024-08-20T21:51:10.7942427Z remote: Counting objects: 85% (1477/1737) 2024-08-20T21:51:10.7942933Z remote: Counting objects: 86% (1494/1737) 2024-08-20T21:51:10.7943459Z remote: Counting objects: 87% (1512/1737) 2024-08-20T21:51:10.7943976Z remote: Counting objects: 88% (1529/1737) 2024-08-20T21:51:10.7944482Z remote: Counting objects: 89% (1546/1737) 2024-08-20T21:51:10.7944999Z remote: Counting objects: 90% (1564/1737) 2024-08-20T21:51:10.7945694Z remote: Counting objects: 91% (1581/1737) 2024-08-20T21:51:10.7946217Z remote: Counting objects: 92% (1599/1737) 2024-08-20T21:51:10.7946722Z remote: Counting objects: 93% (1616/1737) 2024-08-20T21:51:10.7947250Z remote: Counting objects: 94% (1633/1737) 2024-08-20T21:51:10.7947769Z remote: Counting objects: 95% (1651/1737) 2024-08-20T21:51:10.7948291Z remote: Counting objects: 96% (1668/1737) 2024-08-20T21:51:10.7948794Z remote: Counting objects: 97% (1685/1737) 2024-08-20T21:51:10.7949311Z remote: Counting objects: 98% (1703/1737) 2024-08-20T21:51:10.7949828Z remote: Counting objects: 99% (1720/1737) 2024-08-20T21:51:10.7950350Z remote: Counting objects: 100% (1737/1737) 2024-08-20T21:51:10.7950897Z remote: Counting objects: 100% (1737/1737), done. 2024-08-20T21:51:10.7951472Z remote: Compressing objects: 0% (1/898) 2024-08-20T21:51:10.8014914Z remote: Compressing objects: 1% (9/898) 2024-08-20T21:51:10.8131137Z remote: Compressing objects: 2% (18/898) 2024-08-20T21:51:10.8208911Z remote: Compressing objects: 3% (27/898) 2024-08-20T21:51:10.8267819Z remote: Compressing objects: 4% (36/898) 2024-08-20T21:51:10.8310294Z remote: Compressing objects: 5% (45/898) 2024-08-20T21:51:10.8337424Z remote: Compressing objects: 6% (54/898) 2024-08-20T21:51:10.8367242Z remote: Compressing objects: 7% (63/898) 2024-08-20T21:51:10.8406080Z remote: Compressing objects: 8% (72/898) 2024-08-20T21:51:10.8439653Z remote: Compressing objects: 9% (81/898) 2024-08-20T21:51:10.8469072Z remote: Compressing objects: 10% (90/898) 2024-08-20T21:51:10.8481079Z remote: Compressing objects: 11% (99/898) 2024-08-20T21:51:10.8492481Z remote: Compressing objects: 12% (108/898) 2024-08-20T21:51:10.8508059Z remote: Compressing objects: 13% (117/898) 2024-08-20T21:51:10.8524626Z remote: Compressing objects: 14% (126/898) 2024-08-20T21:51:10.8533155Z remote: Compressing objects: 15% (135/898) 2024-08-20T21:51:10.8537161Z remote: Compressing objects: 16% (144/898) 2024-08-20T21:51:10.8542741Z remote: Compressing objects: 17% (153/898) 2024-08-20T21:51:10.8546732Z remote: Compressing objects: 18% (162/898) 2024-08-20T21:51:10.8550669Z remote: Compressing objects: 19% (171/898) 2024-08-20T21:51:10.8552261Z remote: Compressing objects: 20% (180/898) 2024-08-20T21:51:10.8553228Z remote: Compressing objects: 21% (189/898) 2024-08-20T21:51:10.8553768Z remote: Compressing objects: 22% (198/898) 2024-08-20T21:51:10.8555596Z remote: Compressing objects: 23% (207/898) 2024-08-20T21:51:10.8556895Z remote: Compressing objects: 24% (216/898) 2024-08-20T21:51:10.8558433Z remote: Compressing objects: 25% (225/898) 2024-08-20T21:51:10.8561669Z remote: Compressing objects: 26% (234/898) 2024-08-20T21:51:10.8568012Z remote: Compressing objects: 27% (243/898) 2024-08-20T21:51:10.8571351Z remote: Compressing objects: 28% (252/898) 2024-08-20T21:51:10.8574486Z remote: Compressing objects: 29% (261/898) 2024-08-20T21:51:10.8577738Z remote: Compressing objects: 30% (270/898) 2024-08-20T21:51:10.8580265Z remote: Compressing objects: 31% (279/898) 2024-08-20T21:51:10.8582312Z remote: Compressing objects: 32% (288/898) 2024-08-20T21:51:10.8586042Z remote: Compressing objects: 33% (297/898) 2024-08-20T21:51:10.8588744Z remote: Compressing objects: 34% (306/898) 2024-08-20T21:51:10.8593035Z remote: Compressing objects: 35% (315/898) 2024-08-20T21:51:10.8594830Z remote: Compressing objects: 36% (324/898) 2024-08-20T21:51:10.8597685Z remote: Compressing objects: 37% (333/898) 2024-08-20T21:51:10.8600006Z remote: Compressing objects: 38% (342/898) 2024-08-20T21:51:10.8601720Z remote: Compressing objects: 39% (351/898) 2024-08-20T21:51:10.8602769Z remote: Compressing objects: 40% (360/898) 2024-08-20T21:51:10.8603366Z remote: Compressing objects: 41% (369/898) 2024-08-20T21:51:10.8604343Z remote: Compressing objects: 42% (378/898) 2024-08-20T21:51:10.8604894Z remote: Compressing objects: 43% (387/898) 2024-08-20T21:51:10.8606019Z remote: Compressing objects: 44% (396/898) 2024-08-20T21:51:10.8608623Z remote: Compressing objects: 45% (405/898) 2024-08-20T21:51:10.8609545Z remote: Compressing objects: 46% (414/898) 2024-08-20T21:51:10.8610111Z remote: Compressing objects: 47% (423/898) 2024-08-20T21:51:10.8610649Z remote: Compressing objects: 48% (432/898) 2024-08-20T21:51:10.8611173Z remote: Compressing objects: 49% (441/898) 2024-08-20T21:51:10.8611824Z remote: Compressing objects: 50% (449/898) 2024-08-20T21:51:10.8612372Z remote: Compressing objects: 51% (458/898) 2024-08-20T21:51:10.8612926Z remote: Compressing objects: 52% (467/898) 2024-08-20T21:51:10.8613453Z remote: Compressing objects: 53% (476/898) 2024-08-20T21:51:10.8613995Z remote: Compressing objects: 54% (485/898) 2024-08-20T21:51:10.8614590Z remote: Compressing objects: 55% (494/898) 2024-08-20T21:51:10.8615580Z remote: Compressing objects: 56% (503/898) 2024-08-20T21:51:10.8616160Z remote: Compressing objects: 57% (512/898) 2024-08-20T21:51:10.8616680Z remote: Compressing objects: 58% (521/898) 2024-08-20T21:51:10.8617215Z remote: Compressing objects: 59% (530/898) 2024-08-20T21:51:10.8617757Z remote: Compressing objects: 60% (539/898) 2024-08-20T21:51:10.8618280Z remote: Compressing objects: 61% (548/898) 2024-08-20T21:51:10.8618816Z remote: Compressing objects: 62% (557/898) 2024-08-20T21:51:10.8619356Z remote: Compressing objects: 63% (566/898) 2024-08-20T21:51:10.8619899Z remote: Compressing objects: 64% (575/898) 2024-08-20T21:51:10.8620588Z remote: Compressing objects: 65% (584/898) 2024-08-20T21:51:10.8621374Z remote: Compressing objects: 66% (593/898) 2024-08-20T21:51:10.8621918Z remote: Compressing objects: 67% (602/898) 2024-08-20T21:51:10.8622456Z remote: Compressing objects: 68% (611/898) 2024-08-20T21:51:10.8622976Z remote: Compressing objects: 69% (620/898) 2024-08-20T21:51:10.8623520Z remote: Compressing objects: 70% (629/898) 2024-08-20T21:51:10.8624059Z remote: Compressing objects: 71% (638/898) 2024-08-20T21:51:10.8624602Z remote: Compressing objects: 72% (647/898) 2024-08-20T21:51:10.8625127Z remote: Compressing objects: 73% (656/898) 2024-08-20T21:51:10.8625664Z remote: Compressing objects: 74% (665/898) 2024-08-20T21:51:10.8626206Z remote: Compressing objects: 75% (674/898) 2024-08-20T21:51:10.8629311Z remote: Compressing objects: 76% (683/898) 2024-08-20T21:51:10.8636432Z remote: Compressing objects: 77% (692/898) 2024-08-20T21:51:10.8641034Z remote: Compressing objects: 78% (701/898) 2024-08-20T21:51:10.8654121Z remote: Compressing objects: 79% (710/898) 2024-08-20T21:51:10.8655128Z remote: Compressing objects: 80% (719/898) 2024-08-20T21:51:10.8664977Z remote: Compressing objects: 81% (728/898) 2024-08-20T21:51:10.8671189Z remote: Compressing objects: 82% (737/898) 2024-08-20T21:51:10.8674290Z remote: Compressing objects: 83% (746/898) 2024-08-20T21:51:10.8677015Z remote: Compressing objects: 84% (755/898) 2024-08-20T21:51:10.8683560Z remote: Compressing objects: 85% (764/898) 2024-08-20T21:51:10.8688523Z remote: Compressing objects: 86% (773/898) 2024-08-20T21:51:10.8692804Z remote: Compressing objects: 87% (782/898) 2024-08-20T21:51:10.8694704Z remote: Compressing objects: 88% (791/898) 2024-08-20T21:51:10.8698303Z remote: Compressing objects: 89% (800/898) 2024-08-20T21:51:10.8702034Z remote: Compressing objects: 90% (809/898) 2024-08-20T21:51:10.8707325Z remote: Compressing objects: 91% (818/898) 2024-08-20T21:51:10.8710164Z remote: Compressing objects: 92% (827/898) 2024-08-20T21:51:10.8711984Z remote: Compressing objects: 93% (836/898) 2024-08-20T21:51:10.8714160Z remote: Compressing objects: 94% (845/898) 2024-08-20T21:51:10.8716864Z remote: Compressing objects: 95% (854/898) 2024-08-20T21:51:10.8719906Z remote: Compressing objects: 96% (863/898) 2024-08-20T21:51:10.8888192Z remote: Compressing objects: 97% (872/898) 2024-08-20T21:51:10.8889045Z remote: Compressing objects: 98% (881/898) 2024-08-20T21:51:10.8889589Z remote: Compressing objects: 99% (890/898) 2024-08-20T21:51:10.8890147Z remote: Compressing objects: 100% (898/898) 2024-08-20T21:51:10.8890728Z remote: Compressing objects: 100% (898/898), done. 2024-08-20T21:51:10.8948207Z Receiving objects: 0% (1/30997) 2024-08-20T21:51:10.8985425Z Receiving objects: 1% (310/30997) 2024-08-20T21:51:10.9019970Z Receiving objects: 2% (620/30997) 2024-08-20T21:51:10.9053833Z Receiving objects: 3% (930/30997) 2024-08-20T21:51:10.9088835Z Receiving objects: 4% (1240/30997) 2024-08-20T21:51:10.9119179Z Receiving objects: 5% (1550/30997) 2024-08-20T21:51:10.9149568Z Receiving objects: 6% (1860/30997) 2024-08-20T21:51:10.9189228Z Receiving objects: 7% (2170/30997) 2024-08-20T21:51:10.9213799Z Receiving objects: 8% (2480/30997) 2024-08-20T21:51:10.9246829Z Receiving objects: 9% (2790/30997) 2024-08-20T21:51:10.9313046Z Receiving objects: 10% (3100/30997) 2024-08-20T21:51:10.9355199Z Receiving objects: 11% (3410/30997) 2024-08-20T21:51:10.9418163Z Receiving objects: 12% (3720/30997) 2024-08-20T21:51:10.9543202Z Receiving objects: 13% (4030/30997) 2024-08-20T21:51:10.9678223Z Receiving objects: 14% (4340/30997) 2024-08-20T21:51:10.9742624Z Receiving objects: 15% (4650/30997) 2024-08-20T21:51:10.9813095Z Receiving objects: 16% (4960/30997) 2024-08-20T21:51:10.9870354Z Receiving objects: 17% (5270/30997) 2024-08-20T21:51:10.9893591Z Receiving objects: 18% (5580/30997) 2024-08-20T21:51:10.9947866Z Receiving objects: 19% (5890/30997) 2024-08-20T21:51:10.9978268Z Receiving objects: 20% (6200/30997) 2024-08-20T21:51:10.9995916Z Receiving objects: 21% (6510/30997) 2024-08-20T21:51:11.0142443Z Receiving objects: 22% (6820/30997) 2024-08-20T21:51:11.8403555Z Receiving objects: 23% (7130/30997) 2024-08-20T21:51:11.8893447Z Receiving objects: 24% (7440/30997), 43.20 MiB | 86.38 MiB/s 2024-08-20T21:51:12.3134726Z Receiving objects: 24% (7509/30997), 91.38 MiB | 91.37 MiB/s 2024-08-20T21:51:12.3736234Z Receiving objects: 25% (7750/30997), 91.38 MiB | 91.37 MiB/s 2024-08-20T21:51:12.4349823Z Receiving objects: 26% (8060/30997), 91.38 MiB | 91.37 MiB/s 2024-08-20T21:51:12.4945155Z Receiving objects: 27% (8370/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.5541310Z Receiving objects: 28% (8680/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.6150121Z Receiving objects: 29% (8990/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.6751250Z Receiving objects: 30% (9300/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.7353803Z Receiving objects: 31% (9610/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.8893916Z Receiving objects: 32% (9920/30997), 131.90 MiB | 87.93 MiB/s 2024-08-20T21:51:12.9369065Z Receiving objects: 32% (10172/30997), 177.93 MiB | 88.96 MiB/s 2024-08-20T21:51:13.0053236Z Receiving objects: 33% (10230/30997), 177.93 MiB | 88.96 MiB/s 2024-08-20T21:51:13.0169895Z Receiving objects: 34% (10539/30997), 177.93 MiB | 88.96 MiB/s 2024-08-20T21:51:13.2478161Z Receiving objects: 35% (10849/30997), 177.93 MiB | 88.96 MiB/s 2024-08-20T21:51:13.4553283Z Receiving objects: 36% (11159/30997), 177.93 MiB | 88.96 MiB/s 2024-08-20T21:51:13.4569577Z Receiving objects: 37% (11469/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4589710Z Receiving objects: 38% (11779/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4606905Z Receiving objects: 39% (12089/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4615787Z Receiving objects: 40% (12399/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4625809Z Receiving objects: 41% (12709/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4641526Z Receiving objects: 42% (13019/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4651818Z Receiving objects: 43% (13329/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4670558Z Receiving objects: 44% (13639/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4673441Z Receiving objects: 45% (13949/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4677446Z Receiving objects: 46% (14259/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4693120Z Receiving objects: 47% (14569/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4714083Z Receiving objects: 48% (14879/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4722506Z Receiving objects: 49% (15189/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4912267Z Receiving objects: 50% (15499/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4913308Z Receiving objects: 51% (15809/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4919184Z Receiving objects: 52% (16119/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4924532Z Receiving objects: 53% (16429/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4929266Z Receiving objects: 54% (16739/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4947551Z Receiving objects: 55% (17049/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4954891Z Receiving objects: 56% (17359/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4960396Z Receiving objects: 57% (17669/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4973064Z Receiving objects: 58% (17979/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4984061Z Receiving objects: 59% (18289/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.4993146Z Receiving objects: 60% (18599/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5111492Z Receiving objects: 61% (18909/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5112542Z Receiving objects: 62% (19219/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5117741Z Receiving objects: 63% (19529/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5124697Z Receiving objects: 64% (19839/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5131473Z Receiving objects: 65% (20149/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5139320Z Receiving objects: 66% (20459/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5361268Z Receiving objects: 67% (20768/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5733924Z Receiving objects: 68% (21078/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5737287Z Receiving objects: 69% (21388/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5750021Z Receiving objects: 70% (21698/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5759910Z Receiving objects: 71% (22008/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5773037Z Receiving objects: 72% (22318/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5781179Z Receiving objects: 73% (22628/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5798626Z Receiving objects: 74% (22938/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5805021Z Receiving objects: 75% (23248/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5811321Z Receiving objects: 76% (23558/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.5818426Z Receiving objects: 77% (23868/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6022307Z Receiving objects: 78% (24178/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6053834Z Receiving objects: 79% (24488/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6061099Z Receiving objects: 80% (24798/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6071800Z Receiving objects: 81% (25108/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6081577Z Receiving objects: 82% (25418/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6131730Z Receiving objects: 83% (25728/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6246667Z Receiving objects: 84% (26038/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6248339Z Receiving objects: 85% (26348/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6255486Z Receiving objects: 86% (26658/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6259938Z Receiving objects: 87% (26968/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6266656Z Receiving objects: 88% (27278/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6271523Z Receiving objects: 89% (27588/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6377918Z Receiving objects: 90% (27898/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6394974Z Receiving objects: 91% (28208/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6401782Z Receiving objects: 92% (28518/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6417313Z Receiving objects: 93% (28828/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6448806Z Receiving objects: 94% (29138/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6458863Z Receiving objects: 95% (29448/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6465531Z Receiving objects: 96% (29758/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6469926Z Receiving objects: 97% (30068/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6537673Z Receiving objects: 98% (30378/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6562387Z Receiving objects: 99% (30688/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6563473Z remote: Total 30997 (delta 1010), reused 1469 (delta 820), pack-reused 29260 (from 1) 2024-08-20T21:51:13.6576230Z Receiving objects: 100% (30997/30997), 215.25 MiB | 86.13 MiB/s 2024-08-20T21:51:13.6576903Z Receiving objects: 100% (30997/30997), 232.50 MiB | 83.98 MiB/s, done. 2024-08-20T21:51:13.6609554Z Resolving deltas: 0% (0/16723) 2024-08-20T21:51:13.6613360Z Resolving deltas: 1% (168/16723) 2024-08-20T21:51:13.6615636Z Resolving deltas: 2% (335/16723) 2024-08-20T21:51:13.6621005Z Resolving deltas: 3% (504/16723) 2024-08-20T21:51:13.6624855Z Resolving deltas: 4% (671/16723) 2024-08-20T21:51:13.6627422Z Resolving deltas: 5% (839/16723) 2024-08-20T21:51:13.6629849Z Resolving deltas: 6% (1007/16723) 2024-08-20T21:51:13.6633509Z Resolving deltas: 7% (1172/16723) 2024-08-20T21:51:13.6636903Z Resolving deltas: 8% (1338/16723) 2024-08-20T21:51:13.6640322Z Resolving deltas: 9% (1506/16723) 2024-08-20T21:51:13.6643590Z Resolving deltas: 10% (1673/16723) 2024-08-20T21:51:13.6647211Z Resolving deltas: 11% (1840/16723) 2024-08-20T21:51:13.6650571Z Resolving deltas: 12% (2009/16723) 2024-08-20T21:51:13.6654398Z Resolving deltas: 13% (2174/16723) 2024-08-20T21:51:13.6658024Z Resolving deltas: 14% (2342/16723) 2024-08-20T21:51:13.6661514Z Resolving deltas: 15% (2509/16723) 2024-08-20T21:51:13.6665392Z Resolving deltas: 16% (2676/16723) 2024-08-20T21:51:13.6668797Z Resolving deltas: 17% (2843/16723) 2024-08-20T21:51:13.6672251Z Resolving deltas: 18% (3011/16723) 2024-08-20T21:51:13.6676392Z Resolving deltas: 19% (3178/16723) 2024-08-20T21:51:13.6682241Z Resolving deltas: 20% (3345/16723) 2024-08-20T21:51:13.6688237Z Resolving deltas: 21% (3512/16723) 2024-08-20T21:51:13.6697104Z Resolving deltas: 22% (3680/16723) 2024-08-20T21:51:13.6704042Z Resolving deltas: 23% (3847/16723) 2024-08-20T21:51:13.6708157Z Resolving deltas: 24% (4014/16723) 2024-08-20T21:51:13.6714528Z Resolving deltas: 25% (4181/16723) 2024-08-20T21:51:13.6719126Z Resolving deltas: 26% (4348/16723) 2024-08-20T21:51:13.6722904Z Resolving deltas: 27% (4516/16723) 2024-08-20T21:51:13.6733177Z Resolving deltas: 28% (4683/16723) 2024-08-20T21:51:13.6737224Z Resolving deltas: 29% (4850/16723) 2024-08-20T21:51:13.6740493Z Resolving deltas: 30% (5018/16723) 2024-08-20T21:51:13.6743198Z Resolving deltas: 31% (5186/16723) 2024-08-20T21:51:13.6745627Z Resolving deltas: 32% (5353/16723) 2024-08-20T21:51:13.6748199Z Resolving deltas: 33% (5520/16723) 2024-08-20T21:51:13.6751013Z Resolving deltas: 34% (5686/16723) 2024-08-20T21:51:13.6753654Z Resolving deltas: 35% (5855/16723) 2024-08-20T21:51:13.6757356Z Resolving deltas: 36% (6021/16723) 2024-08-20T21:51:13.6760545Z Resolving deltas: 37% (6188/16723) 2024-08-20T21:51:13.6765276Z Resolving deltas: 38% (6355/16723) 2024-08-20T21:51:13.6773133Z Resolving deltas: 39% (6522/16723) 2024-08-20T21:51:13.6778925Z Resolving deltas: 40% (6690/16723) 2024-08-20T21:51:13.6787068Z Resolving deltas: 41% (6857/16723) 2024-08-20T21:51:13.6791053Z Resolving deltas: 42% (7026/16723) 2024-08-20T21:51:13.6794438Z Resolving deltas: 43% (7191/16723) 2024-08-20T21:51:13.6797091Z Resolving deltas: 44% (7359/16723) 2024-08-20T21:51:13.6800026Z Resolving deltas: 45% (7527/16723) 2024-08-20T21:51:13.6803690Z Resolving deltas: 46% (7693/16723) 2024-08-20T21:51:13.6821595Z Resolving deltas: 47% (7860/16723) 2024-08-20T21:51:13.6824699Z Resolving deltas: 48% (8028/16723) 2024-08-20T21:51:13.6829457Z Resolving deltas: 49% (8195/16723) 2024-08-20T21:51:13.6834605Z Resolving deltas: 50% (8362/16723) 2024-08-20T21:51:13.6843777Z Resolving deltas: 51% (8529/16723) 2024-08-20T21:51:13.6849506Z Resolving deltas: 52% (8696/16723) 2024-08-20T21:51:13.6854436Z Resolving deltas: 53% (8864/16723) 2024-08-20T21:51:13.6858313Z Resolving deltas: 54% (9031/16723) 2024-08-20T21:51:13.6861866Z Resolving deltas: 55% (9198/16723) 2024-08-20T21:51:13.6865587Z Resolving deltas: 56% (9365/16723) 2024-08-20T21:51:13.6869427Z Resolving deltas: 57% (9535/16723) 2024-08-20T21:51:13.6873183Z Resolving deltas: 58% (9700/16723) 2024-08-20T21:51:13.6879189Z Resolving deltas: 59% (9867/16723) 2024-08-20T21:51:13.6884797Z Resolving deltas: 60% (10034/16723) 2024-08-20T21:51:13.6891151Z Resolving deltas: 61% (10202/16723) 2024-08-20T21:51:13.6896396Z Resolving deltas: 62% (10369/16723) 2024-08-20T21:51:13.6903294Z Resolving deltas: 63% (10536/16723) 2024-08-20T21:51:13.6910979Z Resolving deltas: 64% (10703/16723) 2024-08-20T21:51:13.6918845Z Resolving deltas: 65% (10870/16723) 2024-08-20T21:51:13.6923694Z Resolving deltas: 66% (11038/16723) 2024-08-20T21:51:13.6928362Z Resolving deltas: 67% (11205/16723) 2024-08-20T21:51:13.6931565Z Resolving deltas: 68% (11372/16723) 2024-08-20T21:51:13.6936708Z Resolving deltas: 69% (11539/16723) 2024-08-20T21:51:13.6941564Z Resolving deltas: 70% (11707/16723) 2024-08-20T21:51:13.6945808Z Resolving deltas: 71% (11874/16723) 2024-08-20T21:51:13.6949426Z Resolving deltas: 72% (12041/16723) 2024-08-20T21:51:13.6955993Z Resolving deltas: 73% (12208/16723) 2024-08-20T21:51:13.6963390Z Resolving deltas: 74% (12376/16723) 2024-08-20T21:51:13.6968198Z Resolving deltas: 75% (12543/16723) 2024-08-20T21:51:13.6972865Z Resolving deltas: 76% (12711/16723) 2024-08-20T21:51:13.6976137Z Resolving deltas: 77% (12877/16723) 2024-08-20T21:51:13.6983231Z Resolving deltas: 78% (13044/16723) 2024-08-20T21:51:13.6989049Z Resolving deltas: 79% (13212/16723) 2024-08-20T21:51:13.6995444Z Resolving deltas: 80% (13379/16723) 2024-08-20T21:51:13.7000391Z Resolving deltas: 81% (13546/16723) 2024-08-20T21:51:13.7006195Z Resolving deltas: 82% (13713/16723) 2024-08-20T21:51:13.7012805Z Resolving deltas: 83% (13881/16723) 2024-08-20T21:51:13.7019105Z Resolving deltas: 84% (14048/16723) 2024-08-20T21:51:13.7022740Z Resolving deltas: 85% (14215/16723) 2024-08-20T21:51:13.7026138Z Resolving deltas: 86% (14383/16723) 2024-08-20T21:51:13.7029556Z Resolving deltas: 87% (14550/16723) 2024-08-20T21:51:13.7033418Z Resolving deltas: 88% (14717/16723) 2024-08-20T21:51:13.7038342Z Resolving deltas: 89% (14884/16723) 2024-08-20T21:51:13.7044074Z Resolving deltas: 90% (15051/16723) 2024-08-20T21:51:13.7048417Z Resolving deltas: 91% (15219/16723) 2024-08-20T21:51:13.7051892Z Resolving deltas: 92% (15386/16723) 2024-08-20T21:51:13.7055566Z Resolving deltas: 93% (15553/16723) 2024-08-20T21:51:13.7061532Z Resolving deltas: 94% (15720/16723) 2024-08-20T21:51:13.7067561Z Resolving deltas: 95% (15887/16723) 2024-08-20T21:51:13.7073285Z Resolving deltas: 96% (16055/16723) 2024-08-20T21:51:13.7079017Z Resolving deltas: 97% (16222/16723) 2024-08-20T21:51:13.7083452Z Resolving deltas: 98% (16389/16723) 2024-08-20T21:51:13.7210679Z Resolving deltas: 99% (16556/16723) 2024-08-20T21:51:13.7211258Z Resolving deltas: 100% (16723/16723) 2024-08-20T21:51:13.7211738Z Resolving deltas: 100% (16723/16723), done. 2024-08-20T21:51:14.6846980Z + pushd torchbench 2024-08-20T21:51:14.6847369Z ~/workspace/torchbench ~/workspace 2024-08-20T21:51:14.6847885Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T21:51:14.7445543Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T21:51:14.7446303Z 2024-08-20T21:51:14.7446727Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T21:51:14.7447928Z changes and commit them, and you can discard any commits you make in this 2024-08-20T21:51:14.7448693Z state without impacting any branches by switching back to a branch. 2024-08-20T21:51:14.7449149Z 2024-08-20T21:51:14.7449455Z If you want to create a new branch to retain commits you create, you may 2024-08-20T21:51:14.7450257Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T21:51:14.7450683Z 2024-08-20T21:51:14.7450879Z git switch -c 2024-08-20T21:51:14.7451144Z 2024-08-20T21:51:14.7451359Z Or undo this operation with: 2024-08-20T21:51:14.7451609Z 2024-08-20T21:51:14.7451713Z git switch - 2024-08-20T21:51:14.7451970Z 2024-08-20T21:51:14.7452299Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T21:51:14.7452795Z 2024-08-20T21:51:14.7452959Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T21:51:14.7453382Z + '[' '' ']' 2024-08-20T21:51:14.7453733Z + python install.py --continue_on_fail 2024-08-20T21:51:16.7547581Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T21:51:31.6657750Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T21:51:33.3139334Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T21:51:35.2170521Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T21:51:36.8919290Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T21:51:36.9218377Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T21:51:43.3202258Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T21:51:45.8095113Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T21:51:48.3295307Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T21:51:50.8358801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T21:51:50.8360552Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T21:51:51.9215524Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T21:51:54.1455602Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T21:51:54.1758630Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T21:52:38.1454678Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T21:52:45.9135392Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T21:52:52.1525229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T21:52:58.1118343Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T21:53:05.5086976Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T21:53:11.6372942Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T21:53:18.3895361Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T21:53:24.8042180Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T21:53:30.9694732Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T21:53:38.3612506Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T21:53:44.4357978Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T21:53:50.7139925Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T21:53:51.9405768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T21:54:24.6200261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T21:54:29.6039914Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T21:54:34.6197114Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:00:22.6096702Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:00:23.7953295Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:00:25.2569600Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:00:31.4494362Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:00:38.2915342Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:00:43.5215147Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:00:50.9813763Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:00:56.6373641Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:01:01.4406168Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:01:07.3717578Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:01:23.3827588Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:01:29.3375825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:01:33.0051305Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:01:37.6719985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:01:44.2220434Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:01:44.2223694Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:01:57.0200416Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:02:00.7035566Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:02:00.7037618Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:02:08.1242026Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:02:09.3764443Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:02:10.6335158Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:02:54.7948934Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:04:22.9916566Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:04:23.0222229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:04:24.5115722Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:04:24.5418228Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:04:24.5705450Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:04:24.5980155Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:04:24.6251895Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:04:24.6521082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:04:45.1930348Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:04:45.1932104Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:04:46.4513825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:04:48.8719709Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:04:48.9030924Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:04:48.9314402Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:04:48.9590917Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:04:48.9861928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:04:49.0134047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:04:54.6460530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:04:56.0949862Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:04:59.4727079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:04:59.5035784Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:04:59.5323567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:04:59.5596724Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:04:59.5869512Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:04:59.6134200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:05:10.4553978Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:05:22.0326609Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:05:22.0638540Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:05:22.0640200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:05:22.0641864Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:05:24.3329561Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:05:25.9137920Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:05:25.9446669Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:05:50.8647788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:06:04.8669348Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:06:09.4492804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:06:15.4984220Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:06:15.5309363Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:06:15.5600741Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:06:15.5883937Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:06:15.6159797Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:06:15.6433192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:06:15.6701192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:06:15.6974219Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:06:25.5449620Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:06:43.5456532Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:06:43.5775865Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:06:45.7937922Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:06:47.7582479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:06:48.1241287Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:06:48.1241955Z Print all dependencies after TorchBench is installed 2024-08-20T22:06:48.1242505Z + python -mpip freeze 2024-08-20T22:06:48.5870858Z absl-py==2.1.0 2024-08-20T22:06:48.5871346Z accelerate==0.33.0 2024-08-20T22:06:48.5871665Z aiohappyeyeballs==2.4.0 2024-08-20T22:06:48.5871986Z aiohttp==3.10.5 2024-08-20T22:06:48.5872272Z aiosignal==1.3.1 2024-08-20T22:06:48.5872562Z alabaster==0.7.13 2024-08-20T22:06:48.5872908Z annotated-types==0.7.0 2024-08-20T22:06:48.5876336Z antlr4-python3-runtime==4.9.3 2024-08-20T22:06:48.5876916Z anyascii==0.3.2 2024-08-20T22:06:48.5877309Z astroid==3.2.4 2024-08-20T22:06:48.5877620Z asttokens==2.4.1 2024-08-20T22:06:48.5877910Z astunparse==1.6.3 2024-08-20T22:06:48.5878271Z async-timeout==4.0.3 2024-08-20T22:06:48.5878591Z attrs==23.1.0 2024-08-20T22:06:48.5878856Z audioread==3.0.1 2024-08-20T22:06:48.5879142Z babel==2.16.0 2024-08-20T22:06:48.5879415Z backcall==0.2.0 2024-08-20T22:06:48.5879711Z beautifulsoup4==4.12.3 2024-08-20T22:06:48.5880838Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:06:48.5882023Z bibtexparser==2.0.0b7 2024-08-20T22:06:48.5882383Z black==24.8.0 2024-08-20T22:06:48.5882658Z blinker==1.8.2 2024-08-20T22:06:48.5882935Z blis==0.7.11 2024-08-20T22:06:48.5883194Z blobfile==2.1.1 2024-08-20T22:06:48.5883477Z bokeh==3.1.1 2024-08-20T22:06:48.5883744Z boto3==1.19.12 2024-08-20T22:06:48.5884009Z botocore==1.22.12 2024-08-20T22:06:48.5884309Z breathe==4.34.0 2024-08-20T22:06:48.5884588Z bs4==0.0.1 2024-08-20T22:06:48.5884856Z cachetools==5.5.0 2024-08-20T22:06:48.5885141Z cardboardlint==1.3.1 2024-08-20T22:06:48.5885454Z catalogue==2.0.10 2024-08-20T22:06:48.5885744Z certifi==2024.7.4 2024-08-20T22:06:48.5886027Z cffi==1.17.0 2024-08-20T22:06:48.5886357Z charset-normalizer==3.3.2 2024-08-20T22:06:48.5886691Z click==8.1.7 2024-08-20T22:06:48.5886961Z clldutils==3.22.2 2024-08-20T22:06:48.5887245Z cloudpathlib==0.18.1 2024-08-20T22:06:48.5887557Z cloudpickle==3.0.0 2024-08-20T22:06:48.5887862Z colorama==0.4.6 2024-08-20T22:06:48.5888142Z colorlog==6.8.2 2024-08-20T22:06:48.5888404Z comm==0.2.2 2024-08-20T22:06:48.5888669Z confection==0.1.5 2024-08-20T22:06:48.5888962Z contourpy==1.1.1 2024-08-20T22:06:48.5889241Z coremltools==5.0b5 2024-08-20T22:06:48.5889545Z cryptography==43.0.0 2024-08-20T22:06:48.5889850Z csvw==3.3.0 2024-08-20T22:06:48.5890113Z cycler==0.12.1 2024-08-20T22:06:48.5890376Z cymem==2.0.8 2024-08-20T22:06:48.5890685Z DALL-E==0.1 2024-08-20T22:06:48.5890953Z datasets==2.21.0 2024-08-20T22:06:48.5891356Z debugpy==1.8.5 2024-08-20T22:06:48.5891638Z decorator==5.1.1 2024-08-20T22:06:48.5891932Z defusedxml==0.7.1 2024-08-20T22:06:48.5892224Z Deprecated==1.2.14 2024-08-20T22:06:48.5892947Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:06:48.5893756Z diffusers==0.20.2 2024-08-20T22:06:48.5894046Z dill==0.3.8 2024-08-20T22:06:48.5894296Z diskcache==5.6.3 2024-08-20T22:06:48.5894581Z distro==1.9.0 2024-08-20T22:06:48.5894859Z dlinfo==1.2.1 2024-08-20T22:06:48.5895132Z dnspython==2.6.1 2024-08-20T22:06:48.5895462Z docker-pycreds==0.4.0 2024-08-20T22:06:48.5895783Z docutils==0.16 2024-08-20T22:06:48.5896067Z dominate==2.9.1 2024-08-20T22:06:48.5896896Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:06:48.5897682Z einops==0.8.0 2024-08-20T22:06:48.5897967Z exceptiongroup==1.2.2 2024-08-20T22:06:48.5898286Z execnet==2.1.1 2024-08-20T22:06:48.5898554Z executing==2.0.1 2024-08-20T22:06:48.5898841Z exhale==0.2.3 2024-08-20T22:06:48.5899117Z expecttest==0.1.6 2024-08-20T22:06:48.5899418Z fastjsonschema==2.20.0 2024-08-20T22:06:48.5899718Z FastNLP==0.6.0 2024-08-20T22:06:48.5900047Z ffmpeg-python==0.2.0 2024-08-20T22:06:48.5900357Z filelock==3.13.1 2024-08-20T22:06:48.5900624Z Flask==3.0.3 2024-08-20T22:06:48.5900896Z flatbuffers==2.0 2024-08-20T22:06:48.5901184Z fonttools==4.53.1 2024-08-20T22:06:48.5901475Z frozenlist==1.4.1 2024-08-20T22:06:48.5901754Z fsspec==2024.6.1 2024-08-20T22:06:48.5902032Z ftfy==6.2.3 2024-08-20T22:06:48.5902660Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:06:48.5903385Z future==1.0.0 2024-08-20T22:06:48.5903656Z fvcore==0.1.5.post20221221 2024-08-20T22:06:48.5903986Z gdown==5.2.0 2024-08-20T22:06:48.5904257Z geojson==2.5.0 2024-08-20T22:06:48.5904520Z ghstack==0.8.0 2024-08-20T22:06:48.5904925Z gitdb==4.0.11 2024-08-20T22:06:48.5905191Z GitPython==3.1.43 2024-08-20T22:06:48.5905529Z google-auth==2.34.0 2024-08-20T22:06:48.5905894Z google-auth-oauthlib==1.0.0 2024-08-20T22:06:48.5906243Z greenlet==3.0.3 2024-08-20T22:06:48.5906510Z grpcio==1.65.5 2024-08-20T22:06:48.5906782Z gym==0.26.2 2024-08-20T22:06:48.5907079Z gym-notices==0.0.8 2024-08-20T22:06:48.5907355Z h5py==3.11.0 2024-08-20T22:06:48.5907619Z higher==0.2.1 2024-08-20T22:06:48.5907941Z huggingface-hub==0.24.6 2024-08-20T22:06:48.5908296Z hydra-core==1.3.2 2024-08-20T22:06:48.5908648Z hypothesis==5.35.1 2024-08-20T22:06:48.5908934Z idna==3.7 2024-08-20T22:06:48.5909188Z imageio==2.35.1 2024-08-20T22:06:48.5909453Z imagesize==1.4.1 2024-08-20T22:06:48.5909761Z importlib_metadata==8.3.0 2024-08-20T22:06:48.5910112Z importlib_resources==6.4.3 2024-08-20T22:06:48.5910448Z inflect==7.3.1 2024-08-20T22:06:48.5910714Z iniconfig==2.0.0 2024-08-20T22:06:48.5911001Z iopath==0.1.9 2024-08-20T22:06:48.5911280Z ipykernel==6.29.5 2024-08-20T22:06:48.5911578Z ipython==8.12.0 2024-08-20T22:06:48.5911845Z isodate==0.6.1 2024-08-20T22:06:48.5912120Z isort==5.13.2 2024-08-20T22:06:48.5912395Z itsdangerous==2.2.0 2024-08-20T22:06:48.5912677Z jedi==0.19.1 2024-08-20T22:06:48.5912942Z Jinja2==3.1.4 2024-08-20T22:06:48.5913212Z jmespath==0.10.0 2024-08-20T22:06:48.5913494Z joblib==1.4.2 2024-08-20T22:06:48.5913758Z jsonpatch==1.33 2024-08-20T22:06:48.5914047Z jsonpointer==3.0.0 2024-08-20T22:06:48.5914345Z jsonschema==4.23.0 2024-08-20T22:06:48.5914736Z jsonschema-specifications==2023.12.1 2024-08-20T22:06:48.5915153Z junitparser==2.1.1 2024-08-20T22:06:48.5915494Z jupyter-cache==0.6.1 2024-08-20T22:06:48.5915809Z jupyter_client==8.6.2 2024-08-20T22:06:48.5916112Z jupyter_core==5.7.2 2024-08-20T22:06:48.5916448Z kaldi-io==0.9.8 2024-08-20T22:06:48.5916732Z kiwisolver==1.4.5 2024-08-20T22:06:48.5917003Z kornia==0.7.3 2024-08-20T22:06:48.5917274Z kornia_rs==0.1.5 2024-08-20T22:06:48.5917556Z lameenc==1.7.0 2024-08-20T22:06:48.5917830Z langcodes==3.4.0 2024-08-20T22:06:48.5918107Z langdetect==1.0.9 2024-08-20T22:06:48.5918450Z language-tags==1.2.0 2024-08-20T22:06:48.5918759Z language_data==1.2.0 2024-08-20T22:06:48.5919044Z lark==0.12.0 2024-08-20T22:06:48.5919306Z libcst==1.1.0 2024-08-20T22:06:48.5919574Z librosa==0.9.2 2024-08-20T22:06:48.5919851Z lintrunner==0.12.5 2024-08-20T22:06:48.5920130Z llvmlite==0.32.1 2024-08-20T22:06:48.5920404Z lxml==4.9.4 2024-08-20T22:06:48.5920703Z marisa-trie==1.2.0 2024-08-20T22:06:48.5920976Z Markdown==3.7 2024-08-20T22:06:48.5921388Z markdown-it-py==2.2.0 2024-08-20T22:06:48.5921700Z MarkupSafe==2.1.5 2024-08-20T22:06:48.5921992Z matplotlib==3.5.3 2024-08-20T22:06:48.5922339Z matplotlib-inline==0.1.7 2024-08-20T22:06:48.5922669Z mccabe==0.7.0 2024-08-20T22:06:48.5922981Z mdit-py-plugins==0.3.5 2024-08-20T22:06:48.5923293Z mdurl==0.1.2 2024-08-20T22:06:48.5923595Z ml-dtypes==0.2.0 2024-08-20T22:06:48.5923868Z MonkeyType==23.3.0 2024-08-20T22:06:48.5924210Z more-itertools==10.4.0 2024-08-20T22:06:48.5924520Z mpmath==1.3.0 2024-08-20T22:06:48.5924799Z multidict==6.0.5 2024-08-20T22:06:48.5925078Z multiprocess==0.70.16 2024-08-20T22:06:48.5925393Z murmurhash==1.0.10 2024-08-20T22:06:48.5925679Z musdb==0.4.2 2024-08-20T22:06:48.5925933Z museval==0.4.1 2024-08-20T22:06:48.5926211Z mypy==1.10.0 2024-08-20T22:06:48.5926531Z mypy-extensions==1.0.0 2024-08-20T22:06:48.5926876Z myst-nb==0.17.2 2024-08-20T22:06:48.5927184Z myst-parser==0.18.1 2024-08-20T22:06:48.5927479Z nbclient==0.7.4 2024-08-20T22:06:48.5927758Z nbformat==5.10.4 2024-08-20T22:06:48.5928083Z nest-asyncio==1.6.0 2024-08-20T22:06:48.5928374Z networkx==2.8.8 2024-08-20T22:06:48.5928649Z ninja==1.10.2 2024-08-20T22:06:48.5928915Z nose==1.3.7 2024-08-20T22:06:48.5929159Z numba==0.49.0 2024-08-20T22:06:48.5929426Z numpy==1.21.2 2024-08-20T22:06:48.5929766Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:06:48.5930190Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:06:48.5930620Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:06:48.5931160Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:06:48.5931735Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:06:48.5932135Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:06:48.5932545Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:06:48.5932974Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:06:48.5933415Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:06:48.5933822Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:06:48.5934222Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:06:48.5934635Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:06:48.5934963Z oauthlib==3.2.2 2024-08-20T22:06:48.5935230Z omegaconf==2.3.0 2024-08-20T22:06:48.5935576Z onnx==1.16.1 2024-08-20T22:06:48.5935861Z onnxscript==0.1.0.dev20240817 2024-08-20T22:06:48.5936191Z opacus==1.5.2 2024-08-20T22:06:48.5936517Z opencv-python==4.8.0.74 2024-08-20T22:06:48.5936877Z opt-einsum==3.3.0 2024-08-20T22:06:48.5937163Z optree==0.12.1 2024-08-20T22:06:48.5937426Z packaging==24.1 2024-08-20T22:06:48.5937706Z pandas==2.0.3 2024-08-20T22:06:48.5937978Z parso==0.8.4 2024-08-20T22:06:48.5938230Z patch==1.16 2024-08-20T22:06:48.5938501Z pathspec==0.12.1 2024-08-20T22:06:48.5938787Z pexpect==4.9.0 2024-08-20T22:06:48.5939068Z phonemizer==3.3.0 2024-08-20T22:06:48.5939349Z pickleshare==0.7.5 2024-08-20T22:06:48.5939645Z pillow==10.3.0 2024-08-20T22:06:48.5939937Z pkgutil_resolve_name==1.3.10 2024-08-20T22:06:48.5940273Z platformdirs==4.2.2 2024-08-20T22:06:48.5940575Z pluggy==1.5.0 2024-08-20T22:06:48.5940846Z pooch==1.8.2 2024-08-20T22:06:48.5941122Z portalocker==2.10.1 2024-08-20T22:06:48.5941412Z preshed==3.0.9 2024-08-20T22:06:48.5941696Z prettytable==3.11.0 2024-08-20T22:06:48.5942009Z prompt_toolkit==3.0.47 2024-08-20T22:06:48.5942317Z protobuf==3.20.2 2024-08-20T22:06:48.5942604Z psutil==6.0.0 2024-08-20T22:06:48.5942890Z ptyprocess==0.7.0 2024-08-20T22:06:48.5943186Z pure_eval==0.2.3 2024-08-20T22:06:48.5943509Z py-cpuinfo==9.0.0 2024-08-20T22:06:48.5943798Z pyaml==24.7.0 2024-08-20T22:06:48.5944065Z pyarrow==17.0.0 2024-08-20T22:06:48.5944334Z pyasn1==0.6.0 2024-08-20T22:06:48.5944613Z pyasn1_modules==0.4.0 2024-08-20T22:06:48.5944932Z pyclipper==1.3.0.post5 2024-08-20T22:06:48.5945438Z pycocotools==2.0.7 2024-08-20T22:06:48.5945734Z pycparser==2.22 2024-08-20T22:06:48.5946035Z pycryptodomex==3.20.0 2024-08-20T22:06:48.5946453Z pydantic==2.8.2 2024-08-20T22:06:48.5946730Z pydantic_core==2.20.1 2024-08-20T22:06:48.5947043Z pydot==3.0.1 2024-08-20T22:06:48.5947311Z pygame==2.6.0 2024-08-20T22:06:48.5947584Z PyGithub==2.3.0 2024-08-20T22:06:48.5947854Z Pygments==2.15.0 2024-08-20T22:06:48.5948141Z PyJWT==2.9.0 2024-08-20T22:06:48.5948411Z pylatexenc==2.10 2024-08-20T22:06:48.5948689Z pylint==3.2.6 2024-08-20T22:06:48.5948958Z PyNaCl==1.5.0 2024-08-20T22:06:48.5949226Z pynvml==11.5.3 2024-08-20T22:06:48.5949504Z pyparsing==3.1.2 2024-08-20T22:06:48.5949784Z pypdfium2==4.30.0 2024-08-20T22:06:48.5950069Z pysbd==0.3.4 2024-08-20T22:06:48.5950335Z PySocks==1.7.1 2024-08-20T22:06:48.5950607Z pytest==7.3.2 2024-08-20T22:06:48.5950946Z pytest-benchmark==4.0.0 2024-08-20T22:06:48.5951310Z pytest-cpp==2.3.0 2024-08-20T22:06:48.5951665Z pytest-flakefinder==1.1.0 2024-08-20T22:06:48.5952049Z pytest-rerunfailures==14.0 2024-08-20T22:06:48.5952431Z pytest-xdist==3.3.1 2024-08-20T22:06:48.5952793Z python-dateutil==2.9.0.post0 2024-08-20T22:06:48.5953604Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:06:48.5954366Z python-etcd==0.4.5 2024-08-20T22:06:48.5955439Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:06:48.5957091Z -e git+https://github.com/pytorch/pytorch_sphinx_theme.git@05ee809ecf4c396cce8f44a172484f2946eb255a#egg=pytorch_sphinx_theme 2024-08-20T22:06:48.5958089Z pytz==2024.1 2024-08-20T22:06:48.5958453Z PyWavelets==1.4.1 2024-08-20T22:06:48.5958788Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:06:48.5959201Z pyzmq==26.1.1 2024-08-20T22:06:48.5959457Z rapidfuzz==3.9.6 2024-08-20T22:06:48.5959707Z rdflib==7.0.0 2024-08-20T22:06:48.5959953Z redis==5.0.8 2024-08-20T22:06:48.5960374Z referencing==0.35.1 2024-08-20T22:06:48.5960828Z regex==2024.7.24 2024-08-20T22:06:48.5961105Z requests==2.32.3 2024-08-20T22:06:48.5961456Z requests-oauthlib==2.0.0 2024-08-20T22:06:48.5961783Z resampy==0.3.1 2024-08-20T22:06:48.5962033Z rfc3986==1.5.0 2024-08-20T22:06:48.5962295Z rich==13.7.1 2024-08-20T22:06:48.5962552Z rockset==1.0.3 2024-08-20T22:06:48.5962841Z rpds-py==0.20.0 2024-08-20T22:06:48.5963110Z rsa==4.9 2024-08-20T22:06:48.5963358Z s3transfer==0.5.2 2024-08-20T22:06:48.5963642Z safetensors==0.4.4 2024-08-20T22:06:48.5964043Z scikit-image==0.19.3 2024-08-20T22:06:48.5964378Z scikit-learn==1.3.2 2024-08-20T22:06:48.5964671Z scipy==1.10.1 2024-08-20T22:06:48.5964925Z segments==2.2.1 2024-08-20T22:06:48.5965210Z sentencepiece==0.2.0 2024-08-20T22:06:48.5965548Z sentry-sdk==2.13.0 2024-08-20T22:06:48.5965847Z setproctitle==1.3.3 2024-08-20T22:06:48.5966123Z shapely==2.0.6 2024-08-20T22:06:48.5966398Z shellingham==1.5.4 2024-08-20T22:06:48.5966690Z simplejson==3.19.3 2024-08-20T22:06:48.5967050Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:06:48.5967541Z smart-open==7.0.4 2024-08-20T22:06:48.5967821Z smmap==5.0.1 2024-08-20T22:06:48.5968099Z snowballstemmer==2.2.0 2024-08-20T22:06:48.5968411Z sortedcontainers==2.4.0 2024-08-20T22:06:48.5968730Z soundfile==0.12.1 2024-08-20T22:06:48.5969013Z soupsieve==2.6 2024-08-20T22:06:48.5969280Z spacy==3.7.6 2024-08-20T22:06:48.5969572Z spacy-legacy==3.0.12 2024-08-20T22:06:48.5969911Z spacy-loggers==1.0.5 2024-08-20T22:06:48.5970209Z Sphinx==5.3.0 2024-08-20T22:06:48.5970522Z sphinx-copybutton==0.5.0 2024-08-20T22:06:48.5970885Z sphinx-panels==0.4.1 2024-08-20T22:06:48.5971381Z sphinxcontrib-applehelp==1.0.4 2024-08-20T22:06:48.5971994Z sphinxcontrib-devhelp==1.0.2 2024-08-20T22:06:48.5972447Z sphinxcontrib-htmlhelp==2.0.1 2024-08-20T22:06:48.5972872Z sphinxcontrib-jsmath==1.0.1 2024-08-20T22:06:48.5973284Z sphinxcontrib-katex==0.8.6 2024-08-20T22:06:48.5973677Z sphinxcontrib-qthelp==1.0.3 2024-08-20T22:06:48.5974127Z sphinxcontrib-serializinghtml==1.1.5 2024-08-20T22:06:48.5974540Z SQLAlchemy==2.0.32 2024-08-20T22:06:48.5974829Z srsly==2.4.8 2024-08-20T22:06:48.5975125Z stack-data==0.6.3 2024-08-20T22:06:48.5975417Z stempeg==0.2.3 2024-08-20T22:06:48.5975696Z submitit==1.5.1 2024-08-20T22:06:48.5975960Z sympy==1.12.1 2024-08-20T22:06:48.5976231Z tabulate==0.9.0 2024-08-20T22:06:48.5976562Z tb-nightly==2.13.0a20230426 2024-08-20T22:06:48.5976903Z tensorboard==2.13.0 2024-08-20T22:06:48.5977261Z tensorboard-data-server==0.7.2 2024-08-20T22:06:48.5977636Z tensorboardX==2.6.2.2 2024-08-20T22:06:48.5977955Z termcolor==2.4.0 2024-08-20T22:06:48.5978239Z thinc==8.2.5 2024-08-20T22:06:48.5978503Z threadpoolctl==3.5.0 2024-08-20T22:06:48.5978805Z tifffile==2023.7.10 2024-08-20T22:06:48.5979648Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:06:48.5980442Z tlparse==0.3.25 2024-08-20T22:06:48.5980730Z tokenizers==0.15.2 2024-08-20T22:06:48.5981019Z tomli==2.0.1 2024-08-20T22:06:48.5981290Z tomlkit==0.13.2 2024-08-20T22:06:48.5982546Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp38-cp38-linux_x86_64.whl#sha256=af3cdc81a563c847a86d810c93c14c6dabe8cf56a0cb55c23bba116e1c3deca6 2024-08-20T22:06:48.5984284Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:06:48.5985391Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:06:48.5986396Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:06:48.5987606Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:06:48.5988681Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:06:48.5989736Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:06:48.5990772Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:06:48.5991438Z tornado==6.4.1 2024-08-20T22:06:48.5991681Z tqdm==4.66.5 2024-08-20T22:06:48.5991937Z traitlets==5.14.3 2024-08-20T22:06:48.5992215Z transformers==4.38.1 2024-08-20T22:06:48.5992488Z treetable==0.2.5 2024-08-20T22:06:48.5993275Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:06:48.5994263Z typeguard==4.3.0 2024-08-20T22:06:48.5994609Z typer==0.12.4 2024-08-20T22:06:48.5994914Z typing-inspect==0.9.0 2024-08-20T22:06:48.5995233Z typing_extensions==4.12.2 2024-08-20T22:06:48.5995553Z tzdata==2024.1 2024-08-20T22:06:48.5995827Z Unidecode==1.3.8 2024-08-20T22:06:48.5996164Z unittest-xml-reporting==3.2.0 2024-08-20T22:06:48.5996516Z uritemplate==4.1.1 2024-08-20T22:06:48.5996802Z urllib3==1.26.19 2024-08-20T22:06:48.5997061Z visdom==0.2.4 2024-08-20T22:06:48.5997323Z wandb==0.17.7 2024-08-20T22:06:48.5997585Z wasabi==1.1.3 2024-08-20T22:06:48.5997847Z wcwidth==0.2.13 2024-08-20T22:06:48.5998104Z weasel==0.4.1 2024-08-20T22:06:48.5998420Z websocket-client==1.8.0 2024-08-20T22:06:48.5998733Z Werkzeug==3.0.3 2024-08-20T22:06:48.5998989Z wrapt==1.16.0 2024-08-20T22:06:48.5999249Z xdoctest==1.1.0 2024-08-20T22:06:48.5999517Z xxhash==3.5.0 2024-08-20T22:06:48.5999792Z xyzservices==2024.6.0 2024-08-20T22:06:48.6000073Z yacs==0.1.8 2024-08-20T22:06:48.6000321Z yarl==1.9.4 2024-08-20T22:06:48.6000611Z z3-solver==4.12.2.0 2024-08-20T22:06:48.6000888Z zipp==3.20.0 2024-08-20T22:06:48.6196977Z + popd 2024-08-20T22:06:48.6197334Z ~/workspace 2024-08-20T22:06:48.6197745Z + [[ cpu_aot_inductor_torchbench_amp_freezing != *cpu* ]] 2024-08-20T22:06:48.6198529Z ++ pwd 2024-08-20T22:06:48.6200540Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T22:06:48.6201043Z + test_dynamo_benchmark torchbench 0 2024-08-20T22:06:48.6205033Z ++ pwd 2024-08-20T22:06:48.6207097Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:06:48.6207657Z + local suite=torchbench 2024-08-20T22:06:48.6207978Z + shift 2024-08-20T22:06:48.6208231Z + local shard_id=0 2024-08-20T22:06:48.6208501Z + shift 2024-08-20T22:06:48.6208901Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *perf_compare* ]] 2024-08-20T22:06:48.6209530Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *perf* ]] 2024-08-20T22:06:48.6210107Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *cpu* ]] 2024-08-20T22:06:48.6210565Z + local dt=float32 2024-08-20T22:06:48.6210956Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *amp* ]] 2024-08-20T22:06:48.6211608Z + dt=amp 2024-08-20T22:06:48.6211999Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *freezing* ]] 2024-08-20T22:06:48.6212842Z + test_single_dynamo_benchmark inference torchbench 0 --inference --amp --freezing 2024-08-20T22:06:48.6213509Z ++ pwd 2024-08-20T22:06:48.6213983Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:06:48.6214668Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:06:48.6227195Z + local name=inference 2024-08-20T22:06:48.6227561Z + shift 2024-08-20T22:06:48.6227836Z + local suite=torchbench 2024-08-20T22:06:48.6228161Z + shift 2024-08-20T22:06:48.6228398Z + local shard_id=0 2024-08-20T22:06:48.6228676Z + shift 2024-08-20T22:06:48.6228970Z + partition_flags=() 2024-08-20T22:06:48.6229276Z + local partition_flags 2024-08-20T22:06:48.6229639Z + [[ -n 2 ]] 2024-08-20T22:06:48.6229933Z + [[ -n 0 ]] 2024-08-20T22:06:48.6230533Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:06:48.6231304Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *perf_compare* ]] 2024-08-20T22:06:48.6231932Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *perf* ]] 2024-08-20T22:06:48.6232553Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *aot_inductor* ]] 2024-08-20T22:06:48.6233233Z + [[ cpu_aot_inductor_torchbench_amp_freezing != *cpu_aot_inductor* ]] 2024-08-20T22:06:48.6234057Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *_avx2* ]] 2024-08-20T22:06:48.6234663Z + [[ cpu_aot_inductor_torchbench_amp_freezing == *_avx512* ]] 2024-08-20T22:06:48.6236581Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --export-aot-inductor --device cpu --inference --amp --freezing --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T22:06:53.7520977Z 2024-08-20T22:06:55.7138327Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:06:55.7140756Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:06:55.7146836Z cpu eval BERT_pytorch 2024-08-20T22:06:56.1046879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:06:56.5117641Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:06:56.9175861Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:06:57.4433729Z ERROR:common: 2024-08-20T22:06:57.4434143Z Traceback (most recent call last): 2024-08-20T22:06:57.4435073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:06:57.4435847Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:06:57.4436657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:06:57.4437604Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:06:57.4438503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:06:57.4439231Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:06:57.4440396Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:06:57.4441266Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:06:57.4442205Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:06:57.4443027Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:06:57.4444041Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:06:57.4444931Z return self._call_impl(*args, **kwargs) 2024-08-20T22:06:57.4446078Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:06:57.4446923Z return forward_call(*args, **kwargs) 2024-08-20T22:06:57.4447808Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:06:57.4448566Z return fn(*args, **kwargs) 2024-08-20T22:06:57.4449518Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:06:57.4450429Z return self._call_impl(*args, **kwargs) 2024-08-20T22:06:57.4451471Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:06:57.4452321Z return forward_call(*args, **kwargs) 2024-08-20T22:06:57.4453250Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:06:57.4454102Z return self._torchdynamo_orig_callable( 2024-08-20T22:06:57.4455050Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:06:57.4455857Z return _compile( 2024-08-20T22:06:57.4456679Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:06:57.4457673Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:06:57.4458773Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:06:57.4459927Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:06:57.4460965Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:06:57.4461792Z return function(*args, **kwargs) 2024-08-20T22:06:57.4462751Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:06:57.4463670Z out_code = transform_code_object(code, transform) 2024-08-20T22:06:57.4464830Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:06:57.4465945Z transformations(instructions, code_options) 2024-08-20T22:06:57.4466873Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:06:57.4467664Z return fn(*args, **kwargs) 2024-08-20T22:06:57.4468558Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:06:57.4469366Z tracer.run() 2024-08-20T22:06:57.4470164Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:06:57.4470963Z super().run() 2024-08-20T22:06:57.4471770Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:06:57.4472566Z while self.step(): 2024-08-20T22:06:57.4473387Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:06:57.4474261Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:06:57.4475243Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:06:57.4476072Z return inner_fn(self, inst) 2024-08-20T22:06:57.4477011Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:06:57.4477890Z self.call_function(fn, args, {}) 2024-08-20T22:06:57.4478860Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:06:57.4479986Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:06:57.4481161Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:06:57.4482142Z return super().call_function(tx, args, kwargs) 2024-08-20T22:06:57.4483195Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:06:57.4484128Z return super().call_function(tx, args, kwargs) 2024-08-20T22:06:57.4485177Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:06:57.4486256Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:06:57.4487540Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:06:57.4488705Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:06:57.4489898Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:06:57.4490822Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:06:57.4492084Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:06:57.4492931Z tracer.run() 2024-08-20T22:06:57.4493740Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:06:57.4494528Z while self.step(): 2024-08-20T22:06:57.4495455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:06:57.4496324Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:06:57.4497312Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:06:57.4498146Z return inner_fn(self, inst) 2024-08-20T22:06:57.4499085Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:06:57.4500030Z self.call_function(fn, args, {}) 2024-08-20T22:06:57.4500998Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:06:57.4502134Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:06:57.4503306Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:06:57.4504339Z return super().call_function(tx, args, kwargs) 2024-08-20T22:06:57.4505368Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:06:57.4506273Z return super().call_function(tx, args, kwargs) 2024-08-20T22:06:57.4507282Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:06:57.4508335Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:06:57.4509581Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:06:57.4510705Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:06:57.4511852Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:06:57.4512754Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:06:57.4513773Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:06:57.4514599Z tracer.run() 2024-08-20T22:06:57.4515381Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:06:57.4516148Z while self.step(): 2024-08-20T22:06:57.4516958Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:06:57.4517804Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:06:57.4518791Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1739, in STORE_ATTR 2024-08-20T22:06:57.4519586Z assert ( 2024-08-20T22:06:57.4519992Z AssertionError: Mutating module attribute mask during export. 2024-08-20T22:06:57.4520402Z 2024-08-20T22:06:57.4520502Z from user code: 2024-08-20T22:06:57.4521362Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 50, in forward 2024-08-20T22:06:57.4522314Z x = transformer.forward(x, mask) 2024-08-20T22:06:57.4523306Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 47, in forward 2024-08-20T22:06:57.4524311Z self.lambda_module.set_mask(mask) 2024-08-20T22:06:57.4525320Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 16, in set_mask 2024-08-20T22:06:57.4526304Z self.mask = mask 2024-08-20T22:06:57.4526505Z 2024-08-20T22:06:57.4526785Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:06:57.4527242Z 2024-08-20T22:06:57.4527526Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:06:57.4528072Z fail_to_run 2024-08-20T22:06:57.4528734Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:06:57.4529410Z TIMING: 2024-08-20T22:06:57.4529966Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:12 | FakeTensorMode.__torch_dispatch__:43 2024-08-20T22:06:57.4530828Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:07:00.4406110Z 2024-08-20T22:07:04.1813522Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:07:04.1814035Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:07:04.1814479Z cpu eval Background_Matting 2024-08-20T22:07:04.1821054Z pass_due_to_skip 2024-08-20T22:07:04.1822272Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:07:04.1822977Z TIMING: 2024-08-20T22:07:04.1823239Z STATS: call_* op count: 0 2024-08-20T22:07:04.1823764Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:07:07.2769936Z 2024-08-20T22:07:09.4013216Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:07:09.4013731Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:07:09.4014181Z cpu eval LearningToPaint 2024-08-20T22:07:09.4943570Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:07:09.5420055Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:07:09.5877168Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:07:36.6364715Z pass 2024-08-20T22:07:36.6366101Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:07:36.6367366Z TIMING: backend_compile:0.0001 entire_frame_compile:0.623 total_wall_time:0.623 2024-08-20T22:07:36.6368943Z STATS: call_* op count: 71 | FakeTensor.__torch_dispatch__:399 | FakeTensorMode.__torch_dispatch__:5946 | ProxyTorchDispatchMode.__torch_dispatch__:1234 2024-08-20T22:07:36.6370494Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique) 2024-08-20T22:07:40.0998233Z 2024-08-20T22:07:41.3851875Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-08-20T22:07:41.4032293Z 2024-08-20T22:07:41.4032393Z 2024-08-20T22:07:41.5036400Z 0% 0.00/528M [00:00::value && (((M < 32) && (M != 16)) || ((N < 32) && (N != 16)))), int>::type = 0; int64_t = long int]’: 2024-08-20T22:12:47.3186067Z /tmp/tmpk7nvzfpp/clhhunrq4n7kt4gvmpqqtyy734a4chbakclns45tfeolz7v2qyps/cf3zpdcr2y64fjjq5kdelkfkf6gbiiyzohy2ctspvtumpsjbgogu.cpp:789:59: required from here 2024-08-20T22:12:47.3188294Z /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/include/ATen/cpu/vec/vec512/vec512_bfloat16.h:1395:27: warning: left shift count >= width of type [-Wshift-count-overflow] 2024-08-20T22:12:47.3189640Z 1395 | __mmask32 dst_mask = (1 << M) - 1; 2024-08-20T22:12:47.3190077Z | ~~~^~~~~ 2024-08-20T22:13:08.1785056Z pass 2024-08-20T22:13:08.1787199Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:13:08.1790524Z TIMING: backend_compile:9e-05 entire_frame_compile:3.5205 total_wall_time:3.5205 2024-08-20T22:13:08.1792180Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:33088 | FakeTensor.__torch_dispatch__:536 | ProxyTorchDispatchMode.__torch_dispatch__:1341 2024-08-20T22:13:08.1793718Z Dynamo produced 1 graphs covering 67 ops with 0 graph breaks (0 unique) 2024-08-20T22:13:11.6968335Z 2024-08-20T22:13:11.9547775Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth 2024-08-20T22:13:11.9742516Z 2024-08-20T22:13:11.9742655Z 2024-08-20T22:13:12.0741412Z 0% 0.00/30.8M [00:00 2024-08-20T22:16:27.4805403Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:16:27.4806398Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:16:27.4807285Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:16:27.4808270Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:16:27.4809129Z return node.target(*args, **kwargs) 2024-08-20T22:16:27.4810185Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:16:27.4811202Z r = self.shape_env.evaluate_expr( 2024-08-20T22:16:27.4812180Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:16:27.4813036Z return retlog(event.run(self)) 2024-08-20T22:16:27.4813959Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:16:27.4814877Z return self.f(*args, **kwargs) 2024-08-20T22:16:27.4815907Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:16:27.4817207Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:16:27.4818656Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:16:27.4819627Z raise self._make_data_dependent_error( 2024-08-20T22:16:27.4820893Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:16:27.4822423Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:16:27.4823060Z 2024-08-20T22:16:27.4823334Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:16:27.4824363Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:16:27.4825158Z return node.target(*args, **kwargs) 2024-08-20T22:16:27.4825445Z 2024-08-20T22:16:27.4825647Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:16:27.4826439Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:16:27.4827411Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:16:27.4828627Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:16:27.4829446Z 2024-08-20T22:16:27.4829591Z User Stack (most recent call last): 2024-08-20T22:16:27.4830013Z (snipped, see stack below for prefix) 2024-08-20T22:16:27.4830990Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:16:27.4831887Z return self.inference(batched_inputs) 2024-08-20T22:16:27.4832889Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:16:27.4833821Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:16:27.4834909Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:16:27.4835838Z images = ImageList.from_tensors( 2024-08-20T22:16:27.4836828Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:16:27.4837917Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:16:27.4838331Z 2024-08-20T22:16:27.4838592Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:16:27.4838998Z 2024-08-20T22:16:27.4839286Z During handling of the above exception, another exception occurred: 2024-08-20T22:16:27.4839744Z 2024-08-20T22:16:27.4839876Z Traceback (most recent call last): 2024-08-20T22:16:27.4840595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:16:27.4841360Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:16:27.4842215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:16:27.4843171Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:16:27.4844036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:16:27.4844767Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:16:27.4845934Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:16:27.4846927Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:16:27.4848022Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:16:27.4848841Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:16:27.4849852Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:16:27.4850763Z return self._call_impl(*args, **kwargs) 2024-08-20T22:16:27.4851805Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:16:27.4852654Z return forward_call(*args, **kwargs) 2024-08-20T22:16:27.4853538Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:16:27.4854369Z return fn(*args, **kwargs) 2024-08-20T22:16:27.4855329Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:16:27.4856227Z return self._call_impl(*args, **kwargs) 2024-08-20T22:16:27.4857191Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:16:27.4858028Z return forward_call(*args, **kwargs) 2024-08-20T22:16:27.4858962Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:16:27.4859800Z return self._torchdynamo_orig_callable( 2024-08-20T22:16:27.4860753Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:16:27.4861561Z return _compile( 2024-08-20T22:16:27.4862407Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:16:27.4863460Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:16:27.4864536Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:16:27.4865459Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:16:27.4866464Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:16:27.4867277Z return function(*args, **kwargs) 2024-08-20T22:16:27.4868197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:16:27.4869098Z out_code = transform_code_object(code, transform) 2024-08-20T22:16:27.4870221Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:16:27.4871209Z transformations(instructions, code_options) 2024-08-20T22:16:27.4872119Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:16:27.4872891Z return fn(*args, **kwargs) 2024-08-20T22:16:27.4873856Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:16:27.4874607Z tracer.run() 2024-08-20T22:16:27.4875349Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:16:27.4876084Z super().run() 2024-08-20T22:16:27.4876939Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:16:27.4877688Z while self.step(): 2024-08-20T22:16:27.4878455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:16:27.4879260Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:16:27.4880172Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:16:27.4880946Z return inner_fn(self, inst) 2024-08-20T22:16:27.4881818Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:16:27.4882703Z self.call_function(fn, args, {}) 2024-08-20T22:16:27.4883607Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:16:27.4884662Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:16:27.4885755Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:16:27.4886629Z return super().call_function(tx, args, kwargs) 2024-08-20T22:16:27.4887608Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:16:27.4888475Z return super().call_function(tx, args, kwargs) 2024-08-20T22:16:27.4889450Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:16:27.4890447Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:16:27.4891915Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:16:27.4893081Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:16:27.4894290Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:16:27.4895213Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:16:27.4896269Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:16:27.4897127Z tracer.run() 2024-08-20T22:16:27.4897938Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:16:27.4898735Z while self.step(): 2024-08-20T22:16:27.4899569Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:16:27.4900435Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:16:27.4901420Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:16:27.4902249Z return inner_fn(self, inst) 2024-08-20T22:16:27.4903194Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:16:27.4904167Z self.call_function(fn, args, {}) 2024-08-20T22:16:27.4905064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:16:27.4906100Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:16:27.4907364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:16:27.4908282Z return super().call_function(tx, args, kwargs) 2024-08-20T22:16:27.4909304Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:16:27.4910208Z return super().call_function(tx, args, kwargs) 2024-08-20T22:16:27.4911290Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:16:27.4912356Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:16:27.4913596Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:16:27.4914730Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:16:27.4915893Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:16:27.4916854Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:16:27.4917879Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:16:27.4918702Z tracer.run() 2024-08-20T22:16:27.4919597Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:16:27.4920333Z while self.step(): 2024-08-20T22:16:27.4921111Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:16:27.4921910Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:16:27.4922817Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:16:27.4923569Z return inner_fn(self, inst) 2024-08-20T22:16:27.4924469Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:16:27.4925311Z self.call_function(fn, args, kwargs) 2024-08-20T22:16:27.4926224Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:16:27.4927256Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:16:27.4928357Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:16:27.4929229Z return super().call_function(tx, args, kwargs) 2024-08-20T22:16:27.4930199Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:16:27.4931272Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:16:27.4932692Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:16:27.4933858Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:16:27.4935049Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:16:27.4935994Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:16:27.4937040Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:16:27.4937889Z tracer.run() 2024-08-20T22:16:27.4938697Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:16:27.4939499Z while self.step(): 2024-08-20T22:16:27.4940321Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:16:27.4941188Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:16:27.4942176Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:16:27.4943018Z return inner_fn(self, inst) 2024-08-20T22:16:27.4944015Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:16:27.4944864Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:16:27.4946369Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:16:27.4947255Z return handler(tx, args, kwargs) 2024-08-20T22:16:27.4948331Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:16:27.4949279Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:16:27.4950266Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:16:27.4951268Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:16:27.4952511Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:16:27.4953576Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:16:27.4954681Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:16:27.4955510Z raise UserError( # noqa: B904 2024-08-20T22:16:27.4958238Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:16:27.4960463Z 2024-08-20T22:16:27.4960721Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:16:27.4961690Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:16:27.4962610Z return node.target(*args, **kwargs) 2024-08-20T22:16:27.4962905Z 2024-08-20T22:16:27.4963091Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:16:27.4963866Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:16:27.4964822Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:16:27.4966028Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:16:27.4966828Z 2024-08-20T22:16:27.4966959Z User Stack (most recent call last): 2024-08-20T22:16:27.4967378Z (snipped, see stack below for prefix) 2024-08-20T22:16:27.4968365Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:16:27.4969240Z return self.inference(batched_inputs) 2024-08-20T22:16:27.4970215Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:16:27.4971188Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:16:27.4972464Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:16:27.4973403Z images = ImageList.from_tensors( 2024-08-20T22:16:27.4974391Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:16:27.4975485Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:16:27.4975911Z 2024-08-20T22:16:27.4976161Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:16:27.4977350Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:16:27.4978173Z 2024-08-20T22:16:27.4978296Z from user code: 2024-08-20T22:16:27.4979165Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:16:27.4980065Z return self.inference(batched_inputs) 2024-08-20T22:16:27.4982000Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:16:27.4982968Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:16:27.4984173Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:16:27.4985036Z images = ImageList.from_tensors( 2024-08-20T22:16:27.4985958Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:16:27.4987049Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:16:27.4987437Z 2024-08-20T22:16:27.4987706Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:16:27.4988141Z 2024-08-20T22:16:27.4988414Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:16:27.4988936Z fail_to_run 2024-08-20T22:16:27.4989507Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:16:27.4990165Z TIMING: 2024-08-20T22:16:27.4990676Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:16:27.4991497Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:16:30.5974469Z 2024-08-20T22:16:46.0934829Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:16:46.0935320Z loading model: 0it [00:15, ?it/s] 2024-08-20T22:16:46.0935763Z cpu eval detectron2_fasterrcnn_r_101_dc5 2024-08-20T22:17:05.1049766Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_101_dc5. Setting accuracy check to cosine 2024-08-20T22:17:05.1618143Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:17:27.0476136Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:17:48.9547077Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:18:00.3374564Z W0820 22:18:00.336545 15835 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:18:00.3376846Z E0820 22:18:00.336705 15835 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:18:00.3729749Z ERROR:common: 2024-08-20T22:18:00.3731187Z Traceback (most recent call last): 2024-08-20T22:18:00.3738360Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:18:00.3739200Z return node.target(*args, **kwargs) 2024-08-20T22:18:00.3740318Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:18:00.3741286Z r = self.shape_env.evaluate_expr( 2024-08-20T22:18:00.3742292Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:18:00.3743153Z return retlog(event.run(self)) 2024-08-20T22:18:00.3744083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:18:00.3744931Z return self.f(*args, **kwargs) 2024-08-20T22:18:00.3746626Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:18:00.3747977Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:18:00.3749420Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:18:00.3750494Z raise self._make_data_dependent_error( 2024-08-20T22:18:00.3752058Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:18:00.3753112Z 2024-08-20T22:18:00.3753387Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:18:00.3754441Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:18:00.3755246Z return node.target(*args, **kwargs) 2024-08-20T22:18:00.3755539Z 2024-08-20T22:18:00.3755745Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:18:00.3756661Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:18:00.3757638Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3758863Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:18:00.3759698Z 2024-08-20T22:18:00.3759857Z User Stack (most recent call last): 2024-08-20T22:18:00.3760272Z (snipped, see stack below for prefix) 2024-08-20T22:18:00.3761275Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:18:00.3762171Z return self.inference(batched_inputs) 2024-08-20T22:18:00.3763187Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:18:00.3764112Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:18:00.3765227Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:18:00.3766163Z images = ImageList.from_tensors( 2024-08-20T22:18:00.3767165Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:18:00.3768265Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:18:00.3768679Z 2024-08-20T22:18:00.3768927Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3769347Z 2024-08-20T22:18:00.3769635Z The above exception was the direct cause of the following exception: 2024-08-20T22:18:00.3770099Z 2024-08-20T22:18:00.3770232Z Traceback (most recent call last): 2024-08-20T22:18:00.3771235Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:18:00.3772051Z ret_val = wrap_fake_exception( 2024-08-20T22:18:00.3773010Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:18:00.3773840Z return fn() 2024-08-20T22:18:00.3774622Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:18:00.3775501Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:18:00.3776494Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:18:00.3777385Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:18:00.3778373Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:18:00.3779166Z return node.target(*args, **kwargs) 2024-08-20T22:18:00.3780211Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:18:00.3781159Z r = self.shape_env.evaluate_expr( 2024-08-20T22:18:00.3782129Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:18:00.3783011Z return retlog(event.run(self)) 2024-08-20T22:18:00.3783923Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:18:00.3784763Z return self.f(*args, **kwargs) 2024-08-20T22:18:00.3785887Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:18:00.3787215Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:18:00.3788654Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:18:00.3789625Z raise self._make_data_dependent_error( 2024-08-20T22:18:00.3791025Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:18:00.3792567Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:18:00.3793212Z 2024-08-20T22:18:00.3793497Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:18:00.3794529Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:18:00.3795335Z return node.target(*args, **kwargs) 2024-08-20T22:18:00.3795640Z 2024-08-20T22:18:00.3795831Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:18:00.3796621Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:18:00.3797578Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3798803Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:18:00.3799638Z 2024-08-20T22:18:00.3799770Z User Stack (most recent call last): 2024-08-20T22:18:00.3800196Z (snipped, see stack below for prefix) 2024-08-20T22:18:00.3801188Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:18:00.3802072Z return self.inference(batched_inputs) 2024-08-20T22:18:00.3803083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:18:00.3804016Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:18:00.3805123Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:18:00.3806039Z images = ImageList.from_tensors( 2024-08-20T22:18:00.3807043Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:18:00.3808143Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:18:00.3808554Z 2024-08-20T22:18:00.3808815Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3809224Z 2024-08-20T22:18:00.3809519Z During handling of the above exception, another exception occurred: 2024-08-20T22:18:00.3809968Z 2024-08-20T22:18:00.3810104Z Traceback (most recent call last): 2024-08-20T22:18:00.3810826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:18:00.3811701Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:18:00.3812505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:18:00.3813446Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:18:00.3814305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:18:00.3815036Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:18:00.3816053Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:18:00.3816927Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:18:00.3817950Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:18:00.3818772Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:18:00.3819796Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:18:00.3820694Z return self._call_impl(*args, **kwargs) 2024-08-20T22:18:00.3821663Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:18:00.3822518Z return forward_call(*args, **kwargs) 2024-08-20T22:18:00.3823480Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:18:00.3824334Z return fn(*args, **kwargs) 2024-08-20T22:18:00.3825269Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:18:00.3826165Z return self._call_impl(*args, **kwargs) 2024-08-20T22:18:00.3827109Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:18:00.3827938Z return forward_call(*args, **kwargs) 2024-08-20T22:18:00.3828843Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:18:00.3829762Z return self._torchdynamo_orig_callable( 2024-08-20T22:18:00.3830650Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:18:00.3831397Z return _compile( 2024-08-20T22:18:00.3832170Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:18:00.3833061Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:18:00.3834083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:18:00.3834967Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:18:00.3835913Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:18:00.3836901Z return function(*args, **kwargs) 2024-08-20T22:18:00.3837844Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:18:00.3838744Z out_code = transform_code_object(code, transform) 2024-08-20T22:18:00.3839857Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:18:00.3840856Z transformations(instructions, code_options) 2024-08-20T22:18:00.3841779Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:18:00.3842552Z return fn(*args, **kwargs) 2024-08-20T22:18:00.3843413Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:18:00.3844208Z tracer.run() 2024-08-20T22:18:00.3845010Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:18:00.3846416Z super().run() 2024-08-20T22:18:00.3847252Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:18:00.3848044Z while self.step(): 2024-08-20T22:18:00.3848892Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:18:00.3849768Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:18:00.3850759Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:18:00.3851676Z return inner_fn(self, inst) 2024-08-20T22:18:00.3852641Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:18:00.3853717Z self.call_function(fn, args, {}) 2024-08-20T22:18:00.3854707Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:18:00.3855826Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:18:00.3857030Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:18:00.3857968Z return super().call_function(tx, args, kwargs) 2024-08-20T22:18:00.3859117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:18:00.3860036Z return super().call_function(tx, args, kwargs) 2024-08-20T22:18:00.3861094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:18:00.3862207Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:18:00.3863599Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:18:00.3864735Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:18:00.3865889Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:18:00.3866805Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:18:00.3867841Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:18:00.3868671Z tracer.run() 2024-08-20T22:18:00.3869454Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:18:00.3870233Z while self.step(): 2024-08-20T22:18:00.3871056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:18:00.3871899Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:18:00.3872845Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:18:00.3873653Z return inner_fn(self, inst) 2024-08-20T22:18:00.3874589Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:18:00.3875443Z self.call_function(fn, args, {}) 2024-08-20T22:18:00.3876567Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:18:00.3877704Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:18:00.3878906Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:18:00.3879848Z return super().call_function(tx, args, kwargs) 2024-08-20T22:18:00.3880901Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:18:00.3881817Z return super().call_function(tx, args, kwargs) 2024-08-20T22:18:00.3882973Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:18:00.3884047Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:18:00.3885298Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:18:00.3886417Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:18:00.3887574Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:18:00.3888484Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:18:00.3889586Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:18:00.3890416Z tracer.run() 2024-08-20T22:18:00.3891284Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:18:00.3892319Z while self.step(): 2024-08-20T22:18:00.3893172Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:18:00.3894119Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:18:00.3895097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:18:00.3895938Z return inner_fn(self, inst) 2024-08-20T22:18:00.3896918Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:18:00.3897835Z self.call_function(fn, args, kwargs) 2024-08-20T22:18:00.3898816Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:18:00.3899948Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:18:00.3901141Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:18:00.3902083Z return super().call_function(tx, args, kwargs) 2024-08-20T22:18:00.3903125Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:18:00.3904296Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:18:00.3905488Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:18:00.3906563Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:18:00.3907665Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:18:00.3908523Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:18:00.3909497Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:18:00.3910279Z tracer.run() 2024-08-20T22:18:00.3911033Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:18:00.3911768Z while self.step(): 2024-08-20T22:18:00.3912551Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:18:00.3913354Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:18:00.3914272Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:18:00.3915035Z return inner_fn(self, inst) 2024-08-20T22:18:00.3915864Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:18:00.3916729Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:18:00.3917752Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:18:00.3918556Z return handler(tx, args, kwargs) 2024-08-20T22:18:00.3919546Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:18:00.3920449Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:18:00.3921360Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:18:00.3922295Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:18:00.3923414Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:18:00.3924425Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:18:00.3925452Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:18:00.3926223Z raise UserError( # noqa: B904 2024-08-20T22:18:00.3928730Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:18:00.3930905Z 2024-08-20T22:18:00.3931283Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:18:00.3932486Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:18:00.3933301Z return node.target(*args, **kwargs) 2024-08-20T22:18:00.3933606Z 2024-08-20T22:18:00.3933799Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:18:00.3934594Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:18:00.3935560Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3936788Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:18:00.3937628Z 2024-08-20T22:18:00.3937764Z User Stack (most recent call last): 2024-08-20T22:18:00.3938195Z (snipped, see stack below for prefix) 2024-08-20T22:18:00.3939181Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:18:00.3940080Z return self.inference(batched_inputs) 2024-08-20T22:18:00.3941091Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:18:00.3942030Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:18:00.3943145Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:18:00.3953880Z images = ImageList.from_tensors( 2024-08-20T22:18:00.3955101Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:18:00.3956234Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:18:00.3956661Z 2024-08-20T22:18:00.3956908Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:18:00.3958220Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:18:00.3959014Z 2024-08-20T22:18:00.3959128Z from user code: 2024-08-20T22:18:00.3959974Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:18:00.3960845Z return self.inference(batched_inputs) 2024-08-20T22:18:00.3961826Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:18:00.3962746Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:18:00.3963814Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:18:00.3964723Z images = ImageList.from_tensors( 2024-08-20T22:18:00.3965695Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:18:00.3966962Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:18:00.3967655Z 2024-08-20T22:18:00.3967959Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:18:00.3968403Z 2024-08-20T22:18:00.3968688Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:18:00.3969234Z fail_to_run 2024-08-20T22:18:00.3969825Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:18:00.3970520Z TIMING: 2024-08-20T22:18:00.3971169Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:18:00.3972349Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:18:03.5512381Z 2024-08-20T22:18:13.1989881Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:18:13.1990470Z loading model: 0it [00:09, ?it/s] 2024-08-20T22:18:13.1991093Z cpu eval detectron2_fasterrcnn_r_101_fpn 2024-08-20T22:18:32.3121649Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_101_fpn. Setting accuracy check to cosine 2024-08-20T22:18:32.3453036Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:18:45.3161325Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:18:58.2143363Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:19:05.1324419Z W0820 22:19:05.131692 16162 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:19:05.1326709Z E0820 22:19:05.131852 16162 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:19:05.1593635Z ERROR:common: 2024-08-20T22:19:05.1593989Z Traceback (most recent call last): 2024-08-20T22:19:05.1596658Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:19:05.1597512Z return node.target(*args, **kwargs) 2024-08-20T22:19:05.1598586Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:19:05.1599535Z r = self.shape_env.evaluate_expr( 2024-08-20T22:19:05.1600514Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:19:05.1601390Z return retlog(event.run(self)) 2024-08-20T22:19:05.1602332Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:19:05.1603293Z return self.f(*args, **kwargs) 2024-08-20T22:19:05.1604342Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:19:05.1605669Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:19:05.1607113Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:19:05.1608082Z raise self._make_data_dependent_error( 2024-08-20T22:19:05.1609473Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:19:05.1610528Z 2024-08-20T22:19:05.1610807Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:19:05.1611964Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:19:05.1612774Z return node.target(*args, **kwargs) 2024-08-20T22:19:05.1613064Z 2024-08-20T22:19:05.1613255Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:19:05.1614401Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:19:05.1615383Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1616593Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:19:05.1617428Z 2024-08-20T22:19:05.1617561Z User Stack (most recent call last): 2024-08-20T22:19:05.1617986Z (snipped, see stack below for prefix) 2024-08-20T22:19:05.1618995Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:19:05.1619998Z return self.inference(batched_inputs) 2024-08-20T22:19:05.1621003Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:19:05.1621939Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:19:05.1623035Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:19:05.1623954Z images = ImageList.from_tensors( 2024-08-20T22:19:05.1624947Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:19:05.1626045Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:19:05.1626460Z 2024-08-20T22:19:05.1626723Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1627129Z 2024-08-20T22:19:05.1627438Z The above exception was the direct cause of the following exception: 2024-08-20T22:19:05.1627889Z 2024-08-20T22:19:05.1628022Z Traceback (most recent call last): 2024-08-20T22:19:05.1628931Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:19:05.1629755Z ret_val = wrap_fake_exception( 2024-08-20T22:19:05.1630688Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:19:05.1631498Z return fn() 2024-08-20T22:19:05.1632267Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:19:05.1633158Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:19:05.1634142Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:19:05.1635011Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:19:05.1636000Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:19:05.1636792Z return node.target(*args, **kwargs) 2024-08-20T22:19:05.1637844Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:19:05.1638770Z r = self.shape_env.evaluate_expr( 2024-08-20T22:19:05.1639745Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:19:05.1640611Z return retlog(event.run(self)) 2024-08-20T22:19:05.1641531Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:19:05.1642365Z return self.f(*args, **kwargs) 2024-08-20T22:19:05.1643381Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:19:05.1644702Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:19:05.1646346Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:19:05.1647313Z raise self._make_data_dependent_error( 2024-08-20T22:19:05.1649944Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:19:05.1651605Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:19:05.1652262Z 2024-08-20T22:19:05.1652534Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:19:05.1653567Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:19:05.1654361Z return node.target(*args, **kwargs) 2024-08-20T22:19:05.1654756Z 2024-08-20T22:19:05.1654947Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:19:05.1655745Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:19:05.1656718Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1657941Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:19:05.1658770Z 2024-08-20T22:19:05.1658904Z User Stack (most recent call last): 2024-08-20T22:19:05.1659332Z (snipped, see stack below for prefix) 2024-08-20T22:19:05.1660330Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:19:05.1661225Z return self.inference(batched_inputs) 2024-08-20T22:19:05.1662224Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:19:05.1663276Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:19:05.1664351Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:19:05.1665262Z images = ImageList.from_tensors( 2024-08-20T22:19:05.1666245Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:19:05.1667399Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:19:05.1667797Z 2024-08-20T22:19:05.1668028Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1668417Z 2024-08-20T22:19:05.1668683Z During handling of the above exception, another exception occurred: 2024-08-20T22:19:05.1669094Z 2024-08-20T22:19:05.1669231Z Traceback (most recent call last): 2024-08-20T22:19:05.1669884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:19:05.1670593Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:19:05.1671329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:19:05.1672214Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:19:05.1673005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:19:05.1673664Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:19:05.1674591Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:19:05.1675413Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:19:05.1676267Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:19:05.1677009Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:19:05.1677948Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:19:05.1678795Z return self._call_impl(*args, **kwargs) 2024-08-20T22:19:05.1679680Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:19:05.1680453Z return forward_call(*args, **kwargs) 2024-08-20T22:19:05.1681331Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:19:05.1682048Z return fn(*args, **kwargs) 2024-08-20T22:19:05.1683143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:19:05.1684013Z return self._call_impl(*args, **kwargs) 2024-08-20T22:19:05.1684935Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:19:05.1685759Z return forward_call(*args, **kwargs) 2024-08-20T22:19:05.1686674Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:19:05.1687556Z return self._torchdynamo_orig_callable( 2024-08-20T22:19:05.1688478Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:19:05.1689260Z return _compile( 2024-08-20T22:19:05.1690077Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:19:05.1691071Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:19:05.1692334Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:19:05.1693283Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:19:05.1694315Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:19:05.1695157Z return function(*args, **kwargs) 2024-08-20T22:19:05.1696103Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:19:05.1697037Z out_code = transform_code_object(code, transform) 2024-08-20T22:19:05.1698194Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:19:05.1699210Z transformations(instructions, code_options) 2024-08-20T22:19:05.1700136Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:19:05.1700926Z return fn(*args, **kwargs) 2024-08-20T22:19:05.1701818Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:19:05.1702627Z tracer.run() 2024-08-20T22:19:05.1703433Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:19:05.1704297Z super().run() 2024-08-20T22:19:05.1705047Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:19:05.1705787Z while self.step(): 2024-08-20T22:19:05.1706548Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:19:05.1707353Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:19:05.1708443Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:19:05.1709254Z return inner_fn(self, inst) 2024-08-20T22:19:05.1710174Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:19:05.1711029Z self.call_function(fn, args, {}) 2024-08-20T22:19:05.1711973Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:19:05.1713080Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:19:05.1714234Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:19:05.1715134Z return super().call_function(tx, args, kwargs) 2024-08-20T22:19:05.1716233Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:19:05.1717153Z return super().call_function(tx, args, kwargs) 2024-08-20T22:19:05.1718177Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:19:05.1719231Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:19:05.1720559Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:19:05.1721710Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:19:05.1722810Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:19:05.1723684Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:19:05.1724647Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:19:05.1725433Z tracer.run() 2024-08-20T22:19:05.1726187Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:19:05.1726929Z while self.step(): 2024-08-20T22:19:05.1727694Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:19:05.1728497Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:19:05.1729608Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:19:05.1730428Z return inner_fn(self, inst) 2024-08-20T22:19:05.1731580Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:19:05.1732468Z self.call_function(fn, args, {}) 2024-08-20T22:19:05.1733452Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:19:05.1734577Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:19:05.1735754Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:19:05.1736696Z return super().call_function(tx, args, kwargs) 2024-08-20T22:19:05.1737746Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:19:05.1738686Z return super().call_function(tx, args, kwargs) 2024-08-20T22:19:05.1739728Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:19:05.1740807Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:19:05.1742096Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:19:05.1743260Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:19:05.1744448Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:19:05.1745550Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:19:05.1746611Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:19:05.1747467Z tracer.run() 2024-08-20T22:19:05.1748276Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:19:05.1749062Z while self.step(): 2024-08-20T22:19:05.1749900Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:19:05.1750771Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:19:05.1751895Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:19:05.1752720Z return inner_fn(self, inst) 2024-08-20T22:19:05.1753705Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:19:05.1754621Z self.call_function(fn, args, kwargs) 2024-08-20T22:19:05.1755606Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:19:05.1756828Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:19:05.1758108Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:19:05.1758980Z return super().call_function(tx, args, kwargs) 2024-08-20T22:19:05.1759961Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:19:05.1760971Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:19:05.1762142Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:19:05.1763215Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:19:05.1764320Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:19:05.1765192Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:19:05.1766165Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:19:05.1766942Z tracer.run() 2024-08-20T22:19:05.1767689Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:19:05.1768434Z while self.step(): 2024-08-20T22:19:05.1769213Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:19:05.1770003Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:19:05.1770915Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:19:05.1771943Z return inner_fn(self, inst) 2024-08-20T22:19:05.1772831Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:19:05.1773761Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:19:05.1774863Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:19:05.1775740Z return handler(tx, args, kwargs) 2024-08-20T22:19:05.1776819Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:19:05.1777771Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:19:05.1778755Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:19:05.1779757Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:19:05.1780907Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:19:05.1781989Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:19:05.1783083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:19:05.1783989Z raise UserError( # noqa: B904 2024-08-20T22:19:05.1786787Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:19:05.1789017Z 2024-08-20T22:19:05.1789280Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:19:05.1790289Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:19:05.1791070Z return node.target(*args, **kwargs) 2024-08-20T22:19:05.1791422Z 2024-08-20T22:19:05.1791606Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:19:05.1792383Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:19:05.1793331Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1794518Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:19:05.1795321Z 2024-08-20T22:19:05.1795450Z User Stack (most recent call last): 2024-08-20T22:19:05.1795865Z (snipped, see stack below for prefix) 2024-08-20T22:19:05.1796829Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:19:05.1797699Z return self.inference(batched_inputs) 2024-08-20T22:19:05.1798662Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:19:05.1799581Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:19:05.1800647Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:19:05.1801555Z images = ImageList.from_tensors( 2024-08-20T22:19:05.1802509Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:19:05.1803672Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:19:05.1804069Z 2024-08-20T22:19:05.1804302Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:19:05.1805402Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:19:05.1806153Z 2024-08-20T22:19:05.1806263Z from user code: 2024-08-20T22:19:05.1807063Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:19:05.1807896Z return self.inference(batched_inputs) 2024-08-20T22:19:05.1808829Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:19:05.1809694Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:19:05.1810693Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:19:05.1811861Z images = ImageList.from_tensors( 2024-08-20T22:19:05.1812869Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:19:05.1813959Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:19:05.1814370Z 2024-08-20T22:19:05.1814676Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:19:05.1815128Z 2024-08-20T22:19:05.1815417Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:19:05.1815980Z fail_to_run 2024-08-20T22:19:05.1816586Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:19:05.1817291Z TIMING: 2024-08-20T22:19:05.1817847Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:51 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:19:05.1818807Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:19:08.2740507Z 2024-08-20T22:19:25.6884414Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:19:25.6884897Z loading model: 0it [00:17, ?it/s] 2024-08-20T22:19:25.6885594Z cpu eval detectron2_fasterrcnn_r_50_c4 2024-08-20T22:19:33.8252123Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_c4. Setting accuracy check to cosine 2024-08-20T22:19:33.8382612Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:20:02.7810548Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:20:31.7194138Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:20:46.5719233Z W0820 22:20:46.571044 16489 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:20:46.5721614Z E0820 22:20:46.571204 16489 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:20:46.5979176Z ERROR:common: 2024-08-20T22:20:46.5979521Z Traceback (most recent call last): 2024-08-20T22:20:46.5982355Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:46.5983202Z return node.target(*args, **kwargs) 2024-08-20T22:20:46.5984258Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:20:46.5985234Z r = self.shape_env.evaluate_expr( 2024-08-20T22:20:46.5986209Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:20:46.5987090Z return retlog(event.run(self)) 2024-08-20T22:20:46.5988023Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:20:46.5988851Z return self.f(*args, **kwargs) 2024-08-20T22:20:46.5989884Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:20:46.5991200Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:20:46.5992638Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:20:46.5993604Z raise self._make_data_dependent_error( 2024-08-20T22:20:46.5994997Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:20:46.5996041Z 2024-08-20T22:20:46.5996317Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:20:46.5997360Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:46.5998166Z return node.target(*args, **kwargs) 2024-08-20T22:20:46.5998454Z 2024-08-20T22:20:46.5998732Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:20:46.5999531Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:20:46.6000486Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6001710Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:20:46.6002545Z 2024-08-20T22:20:46.6002678Z User Stack (most recent call last): 2024-08-20T22:20:46.6003101Z (snipped, see stack below for prefix) 2024-08-20T22:20:46.6004098Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:46.6005176Z return self.inference(batched_inputs) 2024-08-20T22:20:46.6006199Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:46.6007138Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:46.6008234Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:46.6009153Z images = ImageList.from_tensors( 2024-08-20T22:20:46.6010150Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:46.6011491Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:46.6011906Z 2024-08-20T22:20:46.6012169Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6012578Z 2024-08-20T22:20:46.6012880Z The above exception was the direct cause of the following exception: 2024-08-20T22:20:46.6013328Z 2024-08-20T22:20:46.6013468Z Traceback (most recent call last): 2024-08-20T22:20:46.6014380Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:20:46.6015198Z ret_val = wrap_fake_exception( 2024-08-20T22:20:46.6016128Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:20:46.6016932Z return fn() 2024-08-20T22:20:46.6017696Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:20:46.6018583Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:20:46.6019566Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:20:46.6020435Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:20:46.6021416Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:46.6022217Z return node.target(*args, **kwargs) 2024-08-20T22:20:46.6023271Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:20:46.6024201Z r = self.shape_env.evaluate_expr( 2024-08-20T22:20:46.6025166Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:20:46.6026029Z return retlog(event.run(self)) 2024-08-20T22:20:46.6026951Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:20:46.6027774Z return self.f(*args, **kwargs) 2024-08-20T22:20:46.6028806Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:20:46.6030123Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:20:46.6031564Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:20:46.6032532Z raise self._make_data_dependent_error( 2024-08-20T22:20:46.6033785Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:20:46.6035317Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:20:46.6035969Z 2024-08-20T22:20:46.6036236Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:20:46.6037258Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:46.6038060Z return node.target(*args, **kwargs) 2024-08-20T22:20:46.6038345Z 2024-08-20T22:20:46.6038608Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:20:46.6039402Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:20:46.6040383Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6041608Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:20:46.6042439Z 2024-08-20T22:20:46.6042574Z User Stack (most recent call last): 2024-08-20T22:20:46.6042999Z (snipped, see stack below for prefix) 2024-08-20T22:20:46.6044061Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:46.6044948Z return self.inference(batched_inputs) 2024-08-20T22:20:46.6046108Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:46.6047051Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:46.6048156Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:46.6049087Z images = ImageList.from_tensors( 2024-08-20T22:20:46.6050086Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:46.6051235Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:46.6051662Z 2024-08-20T22:20:46.6051913Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6052336Z 2024-08-20T22:20:46.6052624Z During handling of the above exception, another exception occurred: 2024-08-20T22:20:46.6053074Z 2024-08-20T22:20:46.6053221Z Traceback (most recent call last): 2024-08-20T22:20:46.6053928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:20:46.6054696Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:20:46.6055499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:20:46.6056451Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:20:46.6057301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:20:46.6058030Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:20:46.6059039Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:20:46.6059925Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:20:46.6060851Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:20:46.6061654Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:20:46.6062798Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:20:46.6063761Z return self._call_impl(*args, **kwargs) 2024-08-20T22:20:46.6064647Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:20:46.6065418Z return forward_call(*args, **kwargs) 2024-08-20T22:20:46.6066232Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:20:46.6066948Z return fn(*args, **kwargs) 2024-08-20T22:20:46.6067830Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:20:46.6068670Z return self._call_impl(*args, **kwargs) 2024-08-20T22:20:46.6069555Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:20:46.6070337Z return forward_call(*args, **kwargs) 2024-08-20T22:20:46.6071515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:20:46.6072338Z return self._torchdynamo_orig_callable( 2024-08-20T22:20:46.6073265Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:20:46.6074053Z return _compile( 2024-08-20T22:20:46.6074866Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:20:46.6075787Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:20:46.6076857Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:20:46.6077857Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:20:46.6078863Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:20:46.6079672Z return function(*args, **kwargs) 2024-08-20T22:20:46.6080598Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:20:46.6081503Z out_code = transform_code_object(code, transform) 2024-08-20T22:20:46.6082622Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:20:46.6083675Z transformations(instructions, code_options) 2024-08-20T22:20:46.6084536Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:20:46.6085270Z return fn(*args, **kwargs) 2024-08-20T22:20:46.6086088Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:20:46.6086837Z tracer.run() 2024-08-20T22:20:46.6087574Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:20:46.6088312Z super().run() 2024-08-20T22:20:46.6089066Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:46.6089808Z while self.step(): 2024-08-20T22:20:46.6090572Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:46.6091635Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:46.6092638Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:46.6093472Z return inner_fn(self, inst) 2024-08-20T22:20:46.6094409Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:20:46.6095291Z self.call_function(fn, args, {}) 2024-08-20T22:20:46.6096256Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:46.6097391Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:46.6098573Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:20:46.6099496Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:46.6100540Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:46.6101479Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:46.6102532Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:46.6103618Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:46.6104895Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:46.6106142Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:46.6107335Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:46.6108278Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:46.6109322Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:46.6110172Z tracer.run() 2024-08-20T22:20:46.6110987Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:46.6111852Z while self.step(): 2024-08-20T22:20:46.6112770Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:46.6113615Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:46.6114575Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:46.6115385Z return inner_fn(self, inst) 2024-08-20T22:20:46.6116395Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:20:46.6117212Z self.call_function(fn, args, {}) 2024-08-20T22:20:46.6118108Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:46.6119149Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:46.6120237Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:20:46.6121107Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:46.6122083Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:46.6122948Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:46.6124103Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:46.6125165Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:46.6126414Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:46.6127538Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:46.6128691Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:46.6129594Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:46.6130620Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:46.6131710Z tracer.run() 2024-08-20T22:20:46.6132531Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:46.6133316Z while self.step(): 2024-08-20T22:20:46.6134154Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:46.6135022Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:46.6136019Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:46.6136844Z return inner_fn(self, inst) 2024-08-20T22:20:46.6137818Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:20:46.6138730Z self.call_function(fn, args, kwargs) 2024-08-20T22:20:46.6139722Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:20:46.6140852Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:20:46.6142094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:20:46.6143037Z return super().call_function(tx, args, kwargs) 2024-08-20T22:20:46.6144197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:20:46.6145385Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:20:46.6146824Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:20:46.6148085Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:20:46.6149272Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:20:46.6150207Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:20:46.6151264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:20:46.6152101Z tracer.run() 2024-08-20T22:20:46.6152913Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:20:46.6153710Z while self.step(): 2024-08-20T22:20:46.6154550Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:20:46.6155437Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:20:46.6156431Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:20:46.6157263Z return inner_fn(self, inst) 2024-08-20T22:20:46.6158267Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:20:46.6159166Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:20:46.6160251Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:20:46.6161102Z return handler(tx, args, kwargs) 2024-08-20T22:20:46.6162144Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:20:46.6163078Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:20:46.6164019Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:20:46.6164995Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:20:46.6166113Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:20:46.6167167Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:20:46.6168303Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:20:46.6169064Z raise UserError( # noqa: B904 2024-08-20T22:20:46.6171822Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:20:46.6174125Z 2024-08-20T22:20:46.6174395Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:20:46.6175433Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:20:46.6176236Z return node.target(*args, **kwargs) 2024-08-20T22:20:46.6176524Z 2024-08-20T22:20:46.6176837Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:20:46.6177642Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:20:46.6178619Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6179843Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:20:46.6180668Z 2024-08-20T22:20:46.6180814Z User Stack (most recent call last): 2024-08-20T22:20:46.6181226Z (snipped, see stack below for prefix) 2024-08-20T22:20:46.6182284Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:46.6183188Z return self.inference(batched_inputs) 2024-08-20T22:20:46.6184264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:46.6185122Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:46.6186147Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:46.6187008Z images = ImageList.from_tensors( 2024-08-20T22:20:46.6187929Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:46.6188924Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:46.6189319Z 2024-08-20T22:20:46.6189550Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:20:46.6190650Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:20:46.6191394Z 2024-08-20T22:20:46.6191505Z from user code: 2024-08-20T22:20:46.6192315Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:20:46.6193135Z return self.inference(batched_inputs) 2024-08-20T22:20:46.6194064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:20:46.6194926Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:20:46.6195940Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:20:46.6196788Z images = ImageList.from_tensors( 2024-08-20T22:20:46.6197712Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:20:46.6198723Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:20:46.6199104Z 2024-08-20T22:20:46.6199384Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:20:46.6199801Z 2024-08-20T22:20:46.6200085Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:20:46.6200772Z fail_to_run 2024-08-20T22:20:46.6201362Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:20:46.6202056Z TIMING: 2024-08-20T22:20:46.6202596Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:20:46.6203457Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:20:49.6885479Z 2024-08-20T22:21:04.1998902Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:21:04.1999370Z loading model: 0it [00:14, ?it/s] 2024-08-20T22:21:04.1999839Z cpu eval detectron2_fasterrcnn_r_50_dc5 2024-08-20T22:21:18.3569826Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_dc5. Setting accuracy check to cosine 2024-08-20T22:21:18.4134722Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:39.1443766Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:21:59.8735140Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:10.6462734Z W0820 22:22:10.645755 16816 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:22:10.6465044Z E0820 22:22:10.645922 16816 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:22:10.6817820Z ERROR:common: 2024-08-20T22:22:10.6818157Z Traceback (most recent call last): 2024-08-20T22:22:10.6820586Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:10.6821619Z return node.target(*args, **kwargs) 2024-08-20T22:22:10.6823093Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:22:10.6824042Z r = self.shape_env.evaluate_expr( 2024-08-20T22:22:10.6825023Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:22:10.6825905Z return retlog(event.run(self)) 2024-08-20T22:22:10.6826982Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:22:10.6827875Z return self.f(*args, **kwargs) 2024-08-20T22:22:10.6829397Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:22:10.6830776Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:22:10.6832417Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:22:10.6834011Z raise self._make_data_dependent_error( 2024-08-20T22:22:10.6835945Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:10.6837400Z 2024-08-20T22:22:10.6837673Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:10.6838712Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:10.6839518Z return node.target(*args, **kwargs) 2024-08-20T22:22:10.6839811Z 2024-08-20T22:22:10.6840013Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:10.6840791Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:10.6841759Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.6842991Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:10.6843815Z 2024-08-20T22:22:10.6843961Z User Stack (most recent call last): 2024-08-20T22:22:10.6844383Z (snipped, see stack below for prefix) 2024-08-20T22:22:10.6845580Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:10.6846479Z return self.inference(batched_inputs) 2024-08-20T22:22:10.6847490Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:10.6848437Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:10.6849527Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:10.6850457Z images = ImageList.from_tensors( 2024-08-20T22:22:10.6851634Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:10.6852934Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:10.6853371Z 2024-08-20T22:22:10.6853620Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.6854028Z 2024-08-20T22:22:10.6854331Z The above exception was the direct cause of the following exception: 2024-08-20T22:22:10.6854774Z 2024-08-20T22:22:10.6854921Z Traceback (most recent call last): 2024-08-20T22:22:10.6855821Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:22:10.6856734Z ret_val = wrap_fake_exception( 2024-08-20T22:22:10.6857669Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:22:10.6858492Z return fn() 2024-08-20T22:22:10.6859256Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:22:10.6860153Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:22:10.6861135Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:22:10.6862011Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:22:10.6862993Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:10.6863780Z return node.target(*args, **kwargs) 2024-08-20T22:22:10.6864828Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:22:10.6865774Z r = self.shape_env.evaluate_expr( 2024-08-20T22:22:10.6866739Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:22:10.6867591Z return retlog(event.run(self)) 2024-08-20T22:22:10.6868515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:22:10.6869353Z return self.f(*args, **kwargs) 2024-08-20T22:22:10.6870387Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:22:10.6871691Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:22:10.6873125Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:22:10.6874097Z raise self._make_data_dependent_error( 2024-08-20T22:22:10.6875374Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:22:10.6876910Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:10.6877541Z 2024-08-20T22:22:10.6877815Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:10.6878842Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:10.6879635Z return node.target(*args, **kwargs) 2024-08-20T22:22:10.6879917Z 2024-08-20T22:22:10.6880116Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:10.6880891Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:10.6881861Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.6883083Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:10.6883904Z 2024-08-20T22:22:10.6884050Z User Stack (most recent call last): 2024-08-20T22:22:10.6884458Z (snipped, see stack below for prefix) 2024-08-20T22:22:10.6885531Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:10.6886422Z return self.inference(batched_inputs) 2024-08-20T22:22:10.6887423Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:10.6888354Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:10.6889434Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:10.6890463Z images = ImageList.from_tensors( 2024-08-20T22:22:10.6891575Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:10.6892667Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:10.6893080Z 2024-08-20T22:22:10.6893327Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.6893746Z 2024-08-20T22:22:10.6894035Z During handling of the above exception, another exception occurred: 2024-08-20T22:22:10.6894489Z 2024-08-20T22:22:10.6894619Z Traceback (most recent call last): 2024-08-20T22:22:10.6895339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:22:10.6896086Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:22:10.6896880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:22:10.6897827Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:22:10.6898688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:22:10.6899410Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:22:10.6900400Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:22:10.6901287Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:22:10.6902219Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:22:10.6903040Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:22:10.6904045Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:22:10.6905031Z return self._call_impl(*args, **kwargs) 2024-08-20T22:22:10.6905949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:22:10.6906778Z return forward_call(*args, **kwargs) 2024-08-20T22:22:10.6907620Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:22:10.6908366Z return fn(*args, **kwargs) 2024-08-20T22:22:10.6909286Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:22:10.6910174Z return self._call_impl(*args, **kwargs) 2024-08-20T22:22:10.6911089Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:22:10.6911912Z return forward_call(*args, **kwargs) 2024-08-20T22:22:10.6912992Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:22:10.6914111Z return self._torchdynamo_orig_callable( 2024-08-20T22:22:10.6915043Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:22:10.6915851Z return _compile( 2024-08-20T22:22:10.6916680Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:22:10.6917633Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:22:10.6918837Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:22:10.6919771Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:22:10.6920801Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:22:10.6921633Z return function(*args, **kwargs) 2024-08-20T22:22:10.6922589Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:22:10.6923502Z out_code = transform_code_object(code, transform) 2024-08-20T22:22:10.6924718Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:22:10.6925732Z transformations(instructions, code_options) 2024-08-20T22:22:10.6926669Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:22:10.6927448Z return fn(*args, **kwargs) 2024-08-20T22:22:10.6928341Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:22:10.6929147Z tracer.run() 2024-08-20T22:22:10.6929953Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:22:10.6930734Z super().run() 2024-08-20T22:22:10.6931690Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:10.6932496Z while self.step(): 2024-08-20T22:22:10.6933345Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:10.6934205Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:10.6935186Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:10.6936014Z return inner_fn(self, inst) 2024-08-20T22:22:10.6936968Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:22:10.6937844Z self.call_function(fn, args, {}) 2024-08-20T22:22:10.6938799Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:10.6939928Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:10.6941114Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:22:10.6942055Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:10.6943097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:10.6944086Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:10.6945055Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:10.6946444Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:10.6947718Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:10.6948875Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:10.6950064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:10.6951010Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:10.6952060Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:10.6952892Z tracer.run() 2024-08-20T22:22:10.6953702Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:10.6954505Z while self.step(): 2024-08-20T22:22:10.6955481Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:10.6956336Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:10.6957321Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:10.6958250Z return inner_fn(self, inst) 2024-08-20T22:22:10.6959132Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:22:10.6960014Z self.call_function(fn, args, {}) 2024-08-20T22:22:10.6961084Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:10.6962169Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:10.6963324Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:22:10.6964222Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:10.6965236Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:10.6966141Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:10.6967164Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:10.6968227Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:10.6969462Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:10.6970580Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:10.6971978Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:10.6972918Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:10.6973957Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:10.6974801Z tracer.run() 2024-08-20T22:22:10.6975609Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:10.6976403Z while self.step(): 2024-08-20T22:22:10.6977226Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:10.6978092Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:10.6979074Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:10.6979902Z return inner_fn(self, inst) 2024-08-20T22:22:10.6980872Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:22:10.6981770Z self.call_function(fn, args, kwargs) 2024-08-20T22:22:10.6982752Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:22:10.6983981Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:22:10.6985137Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:22:10.6986033Z return super().call_function(tx, args, kwargs) 2024-08-20T22:22:10.6987058Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:22:10.6988119Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:22:10.6989426Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:22:10.7000179Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:22:10.7001418Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:22:10.7002343Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:22:10.7003366Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:22:10.7004202Z tracer.run() 2024-08-20T22:22:10.7005120Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:22:10.7005907Z while self.step(): 2024-08-20T22:22:10.7006720Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:22:10.7007569Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:22:10.7008538Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:22:10.7009347Z return inner_fn(self, inst) 2024-08-20T22:22:10.7010213Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:22:10.7011213Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:22:10.7012486Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:22:10.7013374Z return handler(tx, args, kwargs) 2024-08-20T22:22:10.7014450Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:22:10.7015391Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:22:10.7016377Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:22:10.7017388Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:22:10.7018549Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:22:10.7019629Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:22:10.7020731Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:22:10.7021554Z raise UserError( # noqa: B904 2024-08-20T22:22:10.7024357Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:22:10.7026619Z 2024-08-20T22:22:10.7026888Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:22:10.7027894Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:22:10.7028673Z return node.target(*args, **kwargs) 2024-08-20T22:22:10.7028953Z 2024-08-20T22:22:10.7029139Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:22:10.7029909Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:22:10.7030859Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.7032121Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:22:10.7032882Z 2024-08-20T22:22:10.7033006Z User Stack (most recent call last): 2024-08-20T22:22:10.7033399Z (snipped, see stack below for prefix) 2024-08-20T22:22:10.7034394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:10.7035226Z return self.inference(batched_inputs) 2024-08-20T22:22:10.7036146Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:10.7037015Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:10.7038037Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:10.7039145Z images = ImageList.from_tensors( 2024-08-20T22:22:10.7040097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:10.7041345Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:10.7041775Z 2024-08-20T22:22:10.7042021Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:22:10.7043211Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:22:10.7044024Z 2024-08-20T22:22:10.7044144Z from user code: 2024-08-20T22:22:10.7045010Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:22:10.7046080Z return self.inference(batched_inputs) 2024-08-20T22:22:10.7047095Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:22:10.7048039Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:22:10.7049121Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:22:10.7050054Z images = ImageList.from_tensors( 2024-08-20T22:22:10.7051111Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:22:10.7052209Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:22:10.7052622Z 2024-08-20T22:22:10.7052932Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:22:10.7053381Z 2024-08-20T22:22:10.7053671Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:22:10.7054237Z fail_to_run 2024-08-20T22:22:10.7054844Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:10.7055551Z TIMING: 2024-08-20T22:22:10.7056108Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:22:10.7056990Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:22:13.8142083Z 2024-08-20T22:22:21.5023650Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:22:21.5024156Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:22:21.5024588Z cpu eval detectron2_fasterrcnn_r_50_fpn 2024-08-20T22:22:35.8548920Z WARNING:common:fp64 golden ref were not generated for detectron2_fasterrcnn_r_50_fpn. Setting accuracy check to cosine 2024-08-20T22:22:35.8926462Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:46.6799036Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:22:57.2818396Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:02.9414745Z W0820 22:23:02.940656 17143 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:23:02.9417366Z E0820 22:23:02.940816 17143 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:23:02.9672483Z ERROR:common: 2024-08-20T22:23:02.9673236Z Traceback (most recent call last): 2024-08-20T22:23:02.9674356Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:02.9678499Z return node.target(*args, **kwargs) 2024-08-20T22:23:02.9679743Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:23:02.9680704Z r = self.shape_env.evaluate_expr( 2024-08-20T22:23:02.9681686Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:23:02.9682761Z return retlog(event.run(self)) 2024-08-20T22:23:02.9683696Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:23:02.9684539Z return self.f(*args, **kwargs) 2024-08-20T22:23:02.9685584Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:23:02.9686886Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:23:02.9688340Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:23:02.9689315Z raise self._make_data_dependent_error( 2024-08-20T22:23:02.9690772Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:02.9691942Z 2024-08-20T22:23:02.9692214Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:02.9693258Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:02.9694064Z return node.target(*args, **kwargs) 2024-08-20T22:23:02.9694351Z 2024-08-20T22:23:02.9694540Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:02.9695337Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:02.9696316Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9697544Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:02.9698372Z 2024-08-20T22:23:02.9698520Z User Stack (most recent call last): 2024-08-20T22:23:02.9698932Z (snipped, see stack below for prefix) 2024-08-20T22:23:02.9699934Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:02.9700829Z return self.inference(batched_inputs) 2024-08-20T22:23:02.9701832Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:02.9702749Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:02.9703858Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:02.9704790Z images = ImageList.from_tensors( 2024-08-20T22:23:02.9705786Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:02.9706860Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:02.9707289Z 2024-08-20T22:23:02.9707538Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9707965Z 2024-08-20T22:23:02.9708252Z The above exception was the direct cause of the following exception: 2024-08-20T22:23:02.9708699Z 2024-08-20T22:23:02.9708845Z Traceback (most recent call last): 2024-08-20T22:23:02.9709754Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:23:02.9710563Z ret_val = wrap_fake_exception( 2024-08-20T22:23:02.9711584Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:23:02.9712414Z return fn() 2024-08-20T22:23:02.9713189Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:23:02.9714064Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:23:02.9715048Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:23:02.9716010Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:23:02.9717001Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:02.9717775Z return node.target(*args, **kwargs) 2024-08-20T22:23:02.9718827Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:23:02.9719778Z r = self.shape_env.evaluate_expr( 2024-08-20T22:23:02.9720744Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:23:02.9721600Z return retlog(event.run(self)) 2024-08-20T22:23:02.9722524Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:23:02.9723360Z return self.f(*args, **kwargs) 2024-08-20T22:23:02.9724396Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:23:02.9725712Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:23:02.9727136Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:23:02.9728105Z raise self._make_data_dependent_error( 2024-08-20T22:23:02.9729374Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:23:02.9730903Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:02.9731630Z 2024-08-20T22:23:02.9731916Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:02.9732976Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:02.9733778Z return node.target(*args, **kwargs) 2024-08-20T22:23:02.9734064Z 2024-08-20T22:23:02.9734270Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:02.9735063Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:02.9736022Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9737246Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:02.9738082Z 2024-08-20T22:23:02.9738215Z User Stack (most recent call last): 2024-08-20T22:23:02.9738640Z (snipped, see stack below for prefix) 2024-08-20T22:23:02.9739624Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:02.9740518Z return self.inference(batched_inputs) 2024-08-20T22:23:02.9741523Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:02.9742466Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:02.9743678Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:02.9744578Z images = ImageList.from_tensors( 2024-08-20T22:23:02.9746033Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:02.9747142Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:02.9747553Z 2024-08-20T22:23:02.9747817Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9748223Z 2024-08-20T22:23:02.9748507Z During handling of the above exception, another exception occurred: 2024-08-20T22:23:02.9748965Z 2024-08-20T22:23:02.9749097Z Traceback (most recent call last): 2024-08-20T22:23:02.9749907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:23:02.9750669Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:23:02.9751452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:23:02.9752406Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:23:02.9753274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:23:02.9754004Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:23:02.9755017Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:23:02.9755889Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:23:02.9756816Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:23:02.9757751Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:23:02.9758700Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:23:02.9759529Z return self._call_impl(*args, **kwargs) 2024-08-20T22:23:02.9760409Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:23:02.9761188Z return forward_call(*args, **kwargs) 2024-08-20T22:23:02.9762198Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:23:02.9762933Z return fn(*args, **kwargs) 2024-08-20T22:23:02.9763855Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:23:02.9764736Z return self._call_impl(*args, **kwargs) 2024-08-20T22:23:02.9765671Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:23:02.9766484Z return forward_call(*args, **kwargs) 2024-08-20T22:23:02.9767401Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:23:02.9768232Z return self._torchdynamo_orig_callable( 2024-08-20T22:23:02.9769156Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:23:02.9769926Z return _compile( 2024-08-20T22:23:02.9770742Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:23:02.9771933Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:23:02.9773036Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:23:02.9773986Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:23:02.9775003Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:23:02.9775846Z return function(*args, **kwargs) 2024-08-20T22:23:02.9776806Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:23:02.9777729Z out_code = transform_code_object(code, transform) 2024-08-20T22:23:02.9778939Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:23:02.9779965Z transformations(instructions, code_options) 2024-08-20T22:23:02.9780915Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:23:02.9781710Z return fn(*args, **kwargs) 2024-08-20T22:23:02.9782581Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:23:02.9783404Z tracer.run() 2024-08-20T22:23:02.9784375Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:23:02.9785115Z super().run() 2024-08-20T22:23:02.9785856Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:02.9786594Z while self.step(): 2024-08-20T22:23:02.9787374Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:02.9788182Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:02.9789087Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:02.9789856Z return inner_fn(self, inst) 2024-08-20T22:23:02.9790742Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:23:02.9791731Z self.call_function(fn, args, {}) 2024-08-20T22:23:02.9792665Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:02.9793757Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:02.9794913Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:23:02.9795821Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:02.9796845Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:02.9797739Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:02.9798761Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:02.9799831Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:02.9801071Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:02.9802191Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:02.9803343Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:02.9804332Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:02.9805311Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:02.9806094Z tracer.run() 2024-08-20T22:23:02.9806833Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:02.9807572Z while self.step(): 2024-08-20T22:23:02.9808347Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:02.9809151Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:02.9810056Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:02.9810824Z return inner_fn(self, inst) 2024-08-20T22:23:02.9811984Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:23:02.9812868Z self.call_function(fn, args, {}) 2024-08-20T22:23:02.9813897Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:02.9815025Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:02.9816211Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:23:02.9817152Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:02.9818188Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:02.9819187Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:02.9820239Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:02.9821331Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:02.9822608Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:02.9823856Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:02.9825024Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:02.9825936Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:02.9826960Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:02.9827775Z tracer.run() 2024-08-20T22:23:02.9828561Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:02.9829336Z while self.step(): 2024-08-20T22:23:02.9830148Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:02.9830989Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:02.9831950Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:02.9832865Z return inner_fn(self, inst) 2024-08-20T22:23:02.9833767Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:23:02.9834593Z self.call_function(fn, args, kwargs) 2024-08-20T22:23:02.9835506Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:23:02.9836551Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:23:02.9837651Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:23:02.9838518Z return super().call_function(tx, args, kwargs) 2024-08-20T22:23:02.9839484Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:23:02.9840501Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:23:02.9841679Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:23:02.9842754Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:23:02.9843842Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:23:02.9844714Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:23:02.9846032Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:23:02.9846888Z tracer.run() 2024-08-20T22:23:02.9847820Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:23:02.9848617Z while self.step(): 2024-08-20T22:23:02.9849455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:23:02.9850366Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:23:02.9851433Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:23:02.9852250Z return inner_fn(self, inst) 2024-08-20T22:23:02.9853148Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:23:02.9854167Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:23:02.9855274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:23:02.9856152Z return handler(tx, args, kwargs) 2024-08-20T22:23:02.9857219Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:23:02.9858176Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:23:02.9859163Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:23:02.9860166Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:23:02.9861308Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:23:02.9862402Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:23:02.9863585Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:23:02.9864350Z raise UserError( # noqa: B904 2024-08-20T22:23:02.9866864Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:23:02.9868956Z 2024-08-20T22:23:02.9869207Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:23:02.9870176Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:23:02.9870928Z return node.target(*args, **kwargs) 2024-08-20T22:23:02.9871201Z 2024-08-20T22:23:02.9871392Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:23:02.9872124Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:23:02.9873012Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9874152Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:23:02.9874936Z 2024-08-20T22:23:02.9875060Z User Stack (most recent call last): 2024-08-20T22:23:02.9875457Z (snipped, see stack below for prefix) 2024-08-20T22:23:02.9876369Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:02.9877202Z return self.inference(batched_inputs) 2024-08-20T22:23:02.9878130Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:02.9878998Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:02.9880028Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:02.9880892Z images = ImageList.from_tensors( 2024-08-20T22:23:02.9881859Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:02.9882879Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:02.9883280Z 2024-08-20T22:23:02.9883510Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:23:02.9884613Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:23:02.9885364Z 2024-08-20T22:23:02.9885480Z from user code: 2024-08-20T22:23:02.9886347Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:23:02.9887169Z return self.inference(batched_inputs) 2024-08-20T22:23:02.9888097Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:23:02.9888962Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:23:02.9889966Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:23:02.9890828Z images = ImageList.from_tensors( 2024-08-20T22:23:02.9892022Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:23:02.9893112Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:23:02.9893529Z 2024-08-20T22:23:02.9893834Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:23:02.9894297Z 2024-08-20T22:23:02.9894586Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:23:02.9895150Z fail_to_run 2024-08-20T22:23:02.9895765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:23:02.9896476Z TIMING: 2024-08-20T22:23:02.9897036Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:51 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:23:02.9897922Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:23:06.0682213Z 2024-08-20T22:23:25.2673843Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:23:25.2674316Z loading model: 0it [00:19, ?it/s] 2024-08-20T22:23:25.2676218Z cpu eval detectron2_maskrcnn_r_101_c4 2024-08-20T22:23:38.1403758Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_101_c4. Setting accuracy check to cosine 2024-08-20T22:23:38.1486778Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:09.5508554Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:41.1651482Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:57.3415563Z W0820 22:24:57.340755 17470 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:24:57.3417910Z E0820 22:24:57.340913 17470 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:24:57.3680619Z ERROR:common: 2024-08-20T22:24:57.3681006Z Traceback (most recent call last): 2024-08-20T22:24:57.3682278Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:57.3683374Z return node.target(*args, **kwargs) 2024-08-20T22:24:57.3684476Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:24:57.3685436Z r = self.shape_env.evaluate_expr( 2024-08-20T22:24:57.3686410Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:24:57.3689454Z return retlog(event.run(self)) 2024-08-20T22:24:57.3690651Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:24:57.3691662Z return self.f(*args, **kwargs) 2024-08-20T22:24:57.3693220Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:24:57.3695282Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:24:57.3697658Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:24:57.3698935Z raise self._make_data_dependent_error( 2024-08-20T22:24:57.3700392Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:57.3701446Z 2024-08-20T22:24:57.3701722Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:57.3702761Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:57.3703552Z return node.target(*args, **kwargs) 2024-08-20T22:24:57.3703855Z 2024-08-20T22:24:57.3704044Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:57.3704840Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:57.3705815Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3707040Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:57.3707863Z 2024-08-20T22:24:57.3707995Z User Stack (most recent call last): 2024-08-20T22:24:57.3708426Z (snipped, see stack below for prefix) 2024-08-20T22:24:57.3709442Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:57.3710344Z return self.inference(batched_inputs) 2024-08-20T22:24:57.3711346Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:57.3712288Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:57.3713401Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:57.3714341Z images = ImageList.from_tensors( 2024-08-20T22:24:57.3715337Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:57.3716441Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:57.3716874Z 2024-08-20T22:24:57.3717126Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3717531Z 2024-08-20T22:24:57.3717841Z The above exception was the direct cause of the following exception: 2024-08-20T22:24:57.3718290Z 2024-08-20T22:24:57.3718439Z Traceback (most recent call last): 2024-08-20T22:24:57.3719345Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:24:57.3720175Z ret_val = wrap_fake_exception( 2024-08-20T22:24:57.3721117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:24:57.3721945Z return fn() 2024-08-20T22:24:57.3722714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:24:57.3723606Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:24:57.3724602Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:24:57.3725496Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:24:57.3726588Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:57.3727391Z return node.target(*args, **kwargs) 2024-08-20T22:24:57.3728455Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:24:57.3729405Z r = self.shape_env.evaluate_expr( 2024-08-20T22:24:57.3730369Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:24:57.3731431Z return retlog(event.run(self)) 2024-08-20T22:24:57.3732367Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:24:57.3733208Z return self.f(*args, **kwargs) 2024-08-20T22:24:57.3734249Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:24:57.3735564Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:24:57.3737015Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:24:57.3737984Z raise self._make_data_dependent_error( 2024-08-20T22:24:57.3739260Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:24:57.3740789Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:57.3741440Z 2024-08-20T22:24:57.3741710Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:57.3742740Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:57.3743541Z return node.target(*args, **kwargs) 2024-08-20T22:24:57.3743834Z 2024-08-20T22:24:57.3744039Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:57.3744823Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:57.3746294Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3747526Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:57.3748352Z 2024-08-20T22:24:57.3748510Z User Stack (most recent call last): 2024-08-20T22:24:57.3748918Z (snipped, see stack below for prefix) 2024-08-20T22:24:57.3749923Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:57.3750812Z return self.inference(batched_inputs) 2024-08-20T22:24:57.3751825Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:57.3752757Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:57.3753847Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:57.3754779Z images = ImageList.from_tensors( 2024-08-20T22:24:57.3755776Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:57.3756869Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:57.3757289Z 2024-08-20T22:24:57.3757537Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3758064Z 2024-08-20T22:24:57.3758331Z During handling of the above exception, another exception occurred: 2024-08-20T22:24:57.3758756Z 2024-08-20T22:24:57.3758881Z Traceback (most recent call last): 2024-08-20T22:24:57.3759691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:24:57.3760396Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:24:57.3761142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:24:57.3762029Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:24:57.3762830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:24:57.3763512Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:24:57.3764441Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:24:57.3765393Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:24:57.3766265Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:24:57.3767025Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:24:57.3768147Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:24:57.3769038Z return self._call_impl(*args, **kwargs) 2024-08-20T22:24:57.3769977Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:24:57.3770803Z return forward_call(*args, **kwargs) 2024-08-20T22:24:57.3771925Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:24:57.3772697Z return fn(*args, **kwargs) 2024-08-20T22:24:57.3773664Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:24:57.3774574Z return self._call_impl(*args, **kwargs) 2024-08-20T22:24:57.3775520Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:24:57.3776376Z return forward_call(*args, **kwargs) 2024-08-20T22:24:57.3777320Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:24:57.3778173Z return self._torchdynamo_orig_callable( 2024-08-20T22:24:57.3779118Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:24:57.3779925Z return _compile( 2024-08-20T22:24:57.3780766Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:24:57.3781720Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:24:57.3782823Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:24:57.3783871Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:24:57.3784885Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:24:57.3785785Z return function(*args, **kwargs) 2024-08-20T22:24:57.3786684Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:24:57.3787526Z out_code = transform_code_object(code, transform) 2024-08-20T22:24:57.3788646Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:24:57.3789587Z transformations(instructions, code_options) 2024-08-20T22:24:57.3790463Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:24:57.3791192Z return fn(*args, **kwargs) 2024-08-20T22:24:57.3792020Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:24:57.3792777Z tracer.run() 2024-08-20T22:24:57.3793609Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:24:57.3794344Z super().run() 2024-08-20T22:24:57.3795103Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:57.3795845Z while self.step(): 2024-08-20T22:24:57.3796631Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:57.3797428Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:57.3798351Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:57.3799198Z return inner_fn(self, inst) 2024-08-20T22:24:57.3800093Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:24:57.3800919Z self.call_function(fn, args, {}) 2024-08-20T22:24:57.3801814Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:57.3802871Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:57.3803980Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:24:57.3804860Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:57.3806006Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:57.3806926Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:57.3807966Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:57.3809035Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:57.3810274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:57.3811657Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:57.3812872Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:57.3813817Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:57.3814879Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:57.3815711Z tracer.run() 2024-08-20T22:24:57.3816537Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:57.3817342Z while self.step(): 2024-08-20T22:24:57.3818187Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:57.3819045Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:57.3820045Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:57.3820877Z return inner_fn(self, inst) 2024-08-20T22:24:57.3821844Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:24:57.3822717Z self.call_function(fn, args, {}) 2024-08-20T22:24:57.3823689Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:57.3824816Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:57.3826015Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:24:57.3826948Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:57.3828006Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:57.3829013Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:57.3830066Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:57.3831160Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:57.3832436Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:57.3833701Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:57.3834932Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:57.3835849Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:57.3836864Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:57.3837689Z tracer.run() 2024-08-20T22:24:57.3838489Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:57.3839271Z while self.step(): 2024-08-20T22:24:57.3840084Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:57.3840930Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:57.3841997Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:57.3842775Z return inner_fn(self, inst) 2024-08-20T22:24:57.3843673Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:24:57.3844525Z self.call_function(fn, args, kwargs) 2024-08-20T22:24:57.3845912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:24:57.3847144Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:24:57.3848341Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:24:57.3849268Z return super().call_function(tx, args, kwargs) 2024-08-20T22:24:57.3850328Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:24:57.3851509Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:24:57.3852813Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:24:57.3853964Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:24:57.3855155Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:24:57.3856102Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:24:57.3857160Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:24:57.3858009Z tracer.run() 2024-08-20T22:24:57.3858809Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:24:57.3859614Z while self.step(): 2024-08-20T22:24:57.3860462Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:24:57.3861340Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:24:57.3862322Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:24:57.3863272Z return inner_fn(self, inst) 2024-08-20T22:24:57.3864220Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:24:57.3865208Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:24:57.3866237Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:24:57.3867052Z return handler(tx, args, kwargs) 2024-08-20T22:24:57.3868051Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:24:57.3868943Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:24:57.3869927Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:24:57.3870866Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:24:57.3871939Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:24:57.3872947Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:24:57.3873970Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:24:57.3874719Z raise UserError( # noqa: B904 2024-08-20T22:24:57.3877225Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:24:57.3879343Z 2024-08-20T22:24:57.3879595Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:24:57.3880554Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:24:57.3881299Z return node.target(*args, **kwargs) 2024-08-20T22:24:57.3881571Z 2024-08-20T22:24:57.3881765Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:24:57.3882488Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:24:57.3883393Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3884519Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:24:57.3885273Z 2024-08-20T22:24:57.3885416Z User Stack (most recent call last): 2024-08-20T22:24:57.3885797Z (snipped, see stack below for prefix) 2024-08-20T22:24:57.3886719Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:57.3887552Z return self.inference(batched_inputs) 2024-08-20T22:24:57.3888491Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:57.3889352Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:57.3890381Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:57.3891346Z images = ImageList.from_tensors( 2024-08-20T22:24:57.3892524Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:57.3893626Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:57.3894048Z 2024-08-20T22:24:57.3894298Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:24:57.3895497Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:24:57.3896324Z 2024-08-20T22:24:57.3896433Z from user code: 2024-08-20T22:24:57.3897400Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:24:57.3898284Z return self.inference(batched_inputs) 2024-08-20T22:24:57.3899301Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:24:57.3900242Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:24:57.3901356Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:24:57.3902278Z images = ImageList.from_tensors( 2024-08-20T22:24:57.3903350Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:24:57.3904500Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:24:57.3904882Z 2024-08-20T22:24:57.3905167Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:24:57.3905587Z 2024-08-20T22:24:57.3905881Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:24:57.3906386Z fail_to_run 2024-08-20T22:24:57.3906954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:24:57.3907615Z TIMING: 2024-08-20T22:24:57.3908146Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:24:57.3908946Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:25:00.4682367Z 2024-08-20T22:25:10.4584676Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:25:10.4585358Z loading model: 0it [00:09, ?it/s] 2024-08-20T22:25:10.4586022Z cpu eval detectron2_maskrcnn_r_101_fpn 2024-08-20T22:25:29.2392507Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_101_fpn. Setting accuracy check to cosine 2024-08-20T22:25:29.2713791Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:42.8134310Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:25:56.5009211Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:03.7113386Z W0820 22:26:03.710451 17797 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:26:03.7115690Z E0820 22:26:03.710612 17797 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:26:03.7376021Z ERROR:common: 2024-08-20T22:26:03.7376378Z Traceback (most recent call last): 2024-08-20T22:26:03.7377356Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:03.7378172Z return node.target(*args, **kwargs) 2024-08-20T22:26:03.7382377Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:26:03.7383378Z r = self.shape_env.evaluate_expr( 2024-08-20T22:26:03.7384365Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:26:03.7385228Z return retlog(event.run(self)) 2024-08-20T22:26:03.7386157Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:26:03.7387001Z return self.f(*args, **kwargs) 2024-08-20T22:26:03.7388038Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:26:03.7389347Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:26:03.7390989Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:26:03.7391971Z raise self._make_data_dependent_error( 2024-08-20T22:26:03.7393315Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:03.7394341Z 2024-08-20T22:26:03.7394626Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:03.7395656Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:03.7396562Z return node.target(*args, **kwargs) 2024-08-20T22:26:03.7396862Z 2024-08-20T22:26:03.7397052Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:03.7397840Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:03.7398894Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7400123Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:03.7400945Z 2024-08-20T22:26:03.7401095Z User Stack (most recent call last): 2024-08-20T22:26:03.7401509Z (snipped, see stack below for prefix) 2024-08-20T22:26:03.7402505Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:03.7403401Z return self.inference(batched_inputs) 2024-08-20T22:26:03.7404409Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:03.7405349Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:03.7406435Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:03.7407365Z images = ImageList.from_tensors( 2024-08-20T22:26:03.7408365Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:03.7409452Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:03.7409866Z 2024-08-20T22:26:03.7410113Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7410535Z 2024-08-20T22:26:03.7410824Z The above exception was the direct cause of the following exception: 2024-08-20T22:26:03.7411394Z 2024-08-20T22:26:03.7411530Z Traceback (most recent call last): 2024-08-20T22:26:03.7412453Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:26:03.7413281Z ret_val = wrap_fake_exception( 2024-08-20T22:26:03.7414197Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:26:03.7415016Z return fn() 2024-08-20T22:26:03.7415791Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:26:03.7416680Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:26:03.7417653Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:26:03.7418536Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:26:03.7419519Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:03.7420302Z return node.target(*args, **kwargs) 2024-08-20T22:26:03.7421343Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:26:03.7422289Z r = self.shape_env.evaluate_expr( 2024-08-20T22:26:03.7423264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:26:03.7424137Z return retlog(event.run(self)) 2024-08-20T22:26:03.7425132Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:26:03.7425973Z return self.f(*args, **kwargs) 2024-08-20T22:26:03.7427012Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:26:03.7428327Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:26:03.7429764Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:26:03.7430785Z raise self._make_data_dependent_error( 2024-08-20T22:26:03.7432049Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:26:03.7433584Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:03.7434215Z 2024-08-20T22:26:03.7434494Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:03.7435506Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:03.7436300Z return node.target(*args, **kwargs) 2024-08-20T22:26:03.7436598Z 2024-08-20T22:26:03.7436785Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:03.7437570Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:03.7438548Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7439743Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:03.7440576Z 2024-08-20T22:26:03.7440708Z User Stack (most recent call last): 2024-08-20T22:26:03.7441135Z (snipped, see stack below for prefix) 2024-08-20T22:26:03.7442125Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:03.7443003Z return self.inference(batched_inputs) 2024-08-20T22:26:03.7444004Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:03.7444932Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:03.7446223Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:03.7447154Z images = ImageList.from_tensors( 2024-08-20T22:26:03.7448152Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:03.7449241Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:03.7449652Z 2024-08-20T22:26:03.7449919Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7450323Z 2024-08-20T22:26:03.7450618Z During handling of the above exception, another exception occurred: 2024-08-20T22:26:03.7451136Z 2024-08-20T22:26:03.7451274Z Traceback (most recent call last): 2024-08-20T22:26:03.7451997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:26:03.7452762Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:26:03.7453563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:26:03.7454510Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:26:03.7455365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:26:03.7456088Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:26:03.7457093Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:26:03.7458102Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:26:03.7459035Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:26:03.7459852Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:26:03.7460875Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:26:03.7461783Z return self._call_impl(*args, **kwargs) 2024-08-20T22:26:03.7462736Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:26:03.7463781Z return forward_call(*args, **kwargs) 2024-08-20T22:26:03.7464731Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:26:03.7465443Z return fn(*args, **kwargs) 2024-08-20T22:26:03.7466311Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:26:03.7467153Z return self._call_impl(*args, **kwargs) 2024-08-20T22:26:03.7468225Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:26:03.7469045Z return forward_call(*args, **kwargs) 2024-08-20T22:26:03.7469940Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:26:03.7470770Z return self._torchdynamo_orig_callable( 2024-08-20T22:26:03.7471690Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:26:03.7472468Z return _compile( 2024-08-20T22:26:03.7473266Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:26:03.7474195Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:26:03.7475259Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:26:03.7476176Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:26:03.7477164Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:26:03.7477973Z return function(*args, **kwargs) 2024-08-20T22:26:03.7478899Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:26:03.7479883Z out_code = transform_code_object(code, transform) 2024-08-20T22:26:03.7480942Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:26:03.7481865Z transformations(instructions, code_options) 2024-08-20T22:26:03.7482730Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:26:03.7483470Z return fn(*args, **kwargs) 2024-08-20T22:26:03.7484298Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:26:03.7485034Z tracer.run() 2024-08-20T22:26:03.7485785Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:26:03.7486520Z super().run() 2024-08-20T22:26:03.7487272Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:03.7488005Z while self.step(): 2024-08-20T22:26:03.7488779Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:03.7489577Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:03.7490486Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:03.7491580Z return inner_fn(self, inst) 2024-08-20T22:26:03.7492559Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:26:03.7493442Z self.call_function(fn, args, {}) 2024-08-20T22:26:03.7494413Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:03.7495526Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:03.7496715Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:26:03.7497713Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:03.7498772Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:03.7499700Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:03.7500740Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:03.7501834Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:03.7503117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:03.7504438Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:03.7505528Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:03.7506397Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:03.7507369Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:03.7508152Z tracer.run() 2024-08-20T22:26:03.7508891Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:03.7509630Z while self.step(): 2024-08-20T22:26:03.7510401Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:03.7511195Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:03.7521714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:03.7522558Z return inner_fn(self, inst) 2024-08-20T22:26:03.7523507Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:26:03.7524368Z self.call_function(fn, args, {}) 2024-08-20T22:26:03.7525319Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:03.7526423Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:03.7527603Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:26:03.7528515Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:03.7529531Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:03.7530448Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:03.7531765Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:03.7532874Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:03.7534148Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:03.7535314Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:03.7536631Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:03.7537583Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:03.7538643Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:03.7539483Z tracer.run() 2024-08-20T22:26:03.7540296Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:03.7541101Z while self.step(): 2024-08-20T22:26:03.7542010Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:03.7542865Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:03.7543940Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:03.7544714Z return inner_fn(self, inst) 2024-08-20T22:26:03.7545938Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:26:03.7546835Z self.call_function(fn, args, kwargs) 2024-08-20T22:26:03.7547819Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:26:03.7548940Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:26:03.7550124Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:26:03.7551057Z return super().call_function(tx, args, kwargs) 2024-08-20T22:26:03.7552117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:26:03.7553208Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:26:03.7554489Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:26:03.7555643Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:26:03.7556812Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:26:03.7557749Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:26:03.7558967Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:26:03.7559921Z tracer.run() 2024-08-20T22:26:03.7560693Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:26:03.7561474Z while self.step(): 2024-08-20T22:26:03.7562291Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:26:03.7563143Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:26:03.7564085Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:26:03.7564894Z return inner_fn(self, inst) 2024-08-20T22:26:03.7565758Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:26:03.7566664Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:26:03.7567724Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:26:03.7568584Z return handler(tx, args, kwargs) 2024-08-20T22:26:03.7569623Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:26:03.7570553Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:26:03.7571869Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:26:03.7572866Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:26:03.7574022Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:26:03.7575108Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:26:03.7576204Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:26:03.7577090Z raise UserError( # noqa: B904 2024-08-20T22:26:03.7579813Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:26:03.7582143Z 2024-08-20T22:26:03.7582416Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:26:03.7583456Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:26:03.7584338Z return node.target(*args, **kwargs) 2024-08-20T22:26:03.7584604Z 2024-08-20T22:26:03.7584795Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:26:03.7585518Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:26:03.7586421Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7587549Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:26:03.7588308Z 2024-08-20T22:26:03.7588445Z User Stack (most recent call last): 2024-08-20T22:26:03.7588841Z (snipped, see stack below for prefix) 2024-08-20T22:26:03.7589744Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:03.7590568Z return self.inference(batched_inputs) 2024-08-20T22:26:03.7591495Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:03.7592356Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:03.7593353Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:03.7594218Z images = ImageList.from_tensors( 2024-08-20T22:26:03.7595136Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:03.7596142Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:03.7596522Z 2024-08-20T22:26:03.7596758Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:26:03.7597852Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:26:03.7598615Z 2024-08-20T22:26:03.7598713Z from user code: 2024-08-20T22:26:03.7599516Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:26:03.7600338Z return self.inference(batched_inputs) 2024-08-20T22:26:03.7601250Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:26:03.7602124Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:26:03.7603332Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:26:03.7604232Z images = ImageList.from_tensors( 2024-08-20T22:26:03.7605264Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:26:03.7606334Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:26:03.7606752Z 2024-08-20T22:26:03.7607036Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:26:03.7607475Z 2024-08-20T22:26:03.7607772Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:26:03.7608317Z fail_to_run 2024-08-20T22:26:03.7608891Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:26:03.7609635Z TIMING: 2024-08-20T22:26:03.7610189Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:51 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:26:03.7611131Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:26:06.8538817Z 2024-08-20T22:26:25.0894453Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:26:25.0894917Z loading model: 0it [00:18, ?it/s] 2024-08-20T22:26:25.0895600Z cpu eval detectron2_maskrcnn_r_50_c4 2024-08-20T22:26:33.2772442Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_50_c4. Setting accuracy check to cosine 2024-08-20T22:26:33.2901875Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:04.2671680Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:35.4008528Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:51.3893443Z W0820 22:27:51.388322 18124 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:27:51.3897540Z E0820 22:27:51.388477 18124 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:27:51.4156279Z ERROR:common: 2024-08-20T22:27:51.4157437Z Traceback (most recent call last): 2024-08-20T22:27:51.4158513Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:51.4159331Z return node.target(*args, **kwargs) 2024-08-20T22:27:51.4160394Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:27:51.4161362Z r = self.shape_env.evaluate_expr( 2024-08-20T22:27:51.4162364Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:27:51.4163238Z return retlog(event.run(self)) 2024-08-20T22:27:51.4164158Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:27:51.4165005Z return self.f(*args, **kwargs) 2024-08-20T22:27:51.4166057Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:27:51.4167385Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:27:51.4168835Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:27:51.4169795Z raise self._make_data_dependent_error( 2024-08-20T22:27:51.4171280Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:51.4172337Z 2024-08-20T22:27:51.4172620Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:51.4173673Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:51.4174485Z return node.target(*args, **kwargs) 2024-08-20T22:27:51.4175032Z 2024-08-20T22:27:51.4175233Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:51.4176041Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:51.4177024Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4178254Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:51.4179085Z 2024-08-20T22:27:51.4179238Z User Stack (most recent call last): 2024-08-20T22:27:51.4179880Z (snipped, see stack below for prefix) 2024-08-20T22:27:51.4180892Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:51.4181770Z return self.inference(batched_inputs) 2024-08-20T22:27:51.4182781Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:51.4183723Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:51.4184828Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:51.4185746Z images = ImageList.from_tensors( 2024-08-20T22:27:51.4186746Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:51.4187837Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:51.4188255Z 2024-08-20T22:27:51.4188518Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4188926Z 2024-08-20T22:27:51.4189229Z The above exception was the direct cause of the following exception: 2024-08-20T22:27:51.4189677Z 2024-08-20T22:27:51.4189808Z Traceback (most recent call last): 2024-08-20T22:27:51.4190727Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:27:51.4191542Z ret_val = wrap_fake_exception( 2024-08-20T22:27:51.4192480Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:27:51.4193301Z return fn() 2024-08-20T22:27:51.4194075Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:27:51.4194963Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:27:51.4195949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:27:51.4196830Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:27:51.4197820Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:51.4198613Z return node.target(*args, **kwargs) 2024-08-20T22:27:51.4199672Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:27:51.4200604Z r = self.shape_env.evaluate_expr( 2024-08-20T22:27:51.4201581Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:27:51.4202453Z return retlog(event.run(self)) 2024-08-20T22:27:51.4203378Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:27:51.4204222Z return self.f(*args, **kwargs) 2024-08-20T22:27:51.4205255Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:27:51.4206568Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:27:51.4208002Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:27:51.4209064Z raise self._make_data_dependent_error( 2024-08-20T22:27:51.4210334Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:27:51.4211994Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:51.4212650Z 2024-08-20T22:27:51.4212919Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:51.4214037Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:51.4214837Z return node.target(*args, **kwargs) 2024-08-20T22:27:51.4215124Z 2024-08-20T22:27:51.4215312Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:51.4216107Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:51.4217092Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4218306Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:51.4219127Z 2024-08-20T22:27:51.4219260Z User Stack (most recent call last): 2024-08-20T22:27:51.4219687Z (snipped, see stack below for prefix) 2024-08-20T22:27:51.4220690Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:51.4221593Z return self.inference(batched_inputs) 2024-08-20T22:27:51.4222598Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:51.4223533Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:51.4224636Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:51.4225571Z images = ImageList.from_tensors( 2024-08-20T22:27:51.4226569Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:51.4227651Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:51.4228081Z 2024-08-20T22:27:51.4228328Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4228746Z 2024-08-20T22:27:51.4229033Z During handling of the above exception, another exception occurred: 2024-08-20T22:27:51.4229483Z 2024-08-20T22:27:51.4229631Z Traceback (most recent call last): 2024-08-20T22:27:51.4230338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:27:51.4231100Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:27:51.4231900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:27:51.4232861Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:27:51.4233708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:27:51.4234439Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:27:51.4235454Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:27:51.4236336Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:27:51.4237266Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:27:51.4238076Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:27:51.4239095Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:27:51.4239995Z return self._call_impl(*args, **kwargs) 2024-08-20T22:27:51.4240956Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:27:51.4241853Z return forward_call(*args, **kwargs) 2024-08-20T22:27:51.4242739Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:27:51.4243511Z return fn(*args, **kwargs) 2024-08-20T22:27:51.4244466Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:27:51.4245529Z return self._call_impl(*args, **kwargs) 2024-08-20T22:27:51.4246501Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:27:51.4247506Z return forward_call(*args, **kwargs) 2024-08-20T22:27:51.4248456Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:27:51.4249297Z return self._torchdynamo_orig_callable( 2024-08-20T22:27:51.4250253Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:27:51.4251145Z return _compile( 2024-08-20T22:27:51.4251997Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:27:51.4252940Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:27:51.4254039Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:27:51.4254993Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:27:51.4256033Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:27:51.4256873Z return function(*args, **kwargs) 2024-08-20T22:27:51.4257815Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:27:51.4258736Z out_code = transform_code_object(code, transform) 2024-08-20T22:27:51.4259899Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:27:51.4260915Z transformations(instructions, code_options) 2024-08-20T22:27:51.4261851Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:27:51.4262659Z return fn(*args, **kwargs) 2024-08-20T22:27:51.4263657Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:27:51.4264452Z tracer.run() 2024-08-20T22:27:51.4265237Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:27:51.4266014Z super().run() 2024-08-20T22:27:51.4266812Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:51.4267596Z while self.step(): 2024-08-20T22:27:51.4268408Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:51.4269255Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:51.4270227Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:51.4271031Z return inner_fn(self, inst) 2024-08-20T22:27:51.4271950Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:27:51.4272813Z self.call_function(fn, args, {}) 2024-08-20T22:27:51.4273760Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:51.4274856Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:51.4276125Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:27:51.4277097Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:51.4278094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:51.4278961Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:51.4279949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:51.4280942Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:51.4282195Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:51.4283274Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:51.4284374Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:51.4285241Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:51.4286203Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:51.4286994Z tracer.run() 2024-08-20T22:27:51.4287746Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:51.4288487Z while self.step(): 2024-08-20T22:27:51.4289253Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:51.4290073Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:51.4291128Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:51.4292113Z return inner_fn(self, inst) 2024-08-20T22:27:51.4293064Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:27:51.4293946Z self.call_function(fn, args, {}) 2024-08-20T22:27:51.4294928Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:51.4296057Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:51.4297233Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:27:51.4298175Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:51.4299238Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:51.4300172Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:51.4301221Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:51.4302301Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:51.4303702Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:51.4304850Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:51.4306002Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:51.4306922Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:51.4307959Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:51.4308784Z tracer.run() 2024-08-20T22:27:51.4309564Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:51.4310346Z while self.step(): 2024-08-20T22:27:51.4311331Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:51.4312147Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:51.4313053Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:51.4313833Z return inner_fn(self, inst) 2024-08-20T22:27:51.4314736Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:27:51.4315581Z self.call_function(fn, args, kwargs) 2024-08-20T22:27:51.4316546Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:27:51.4317595Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:27:51.4318700Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:27:51.4319578Z return super().call_function(tx, args, kwargs) 2024-08-20T22:27:51.4320553Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:27:51.4321559Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:27:51.4322749Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:27:51.4323827Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:27:51.4324940Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:27:51.4325797Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:27:51.4326783Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:27:51.4327576Z tracer.run() 2024-08-20T22:27:51.4328340Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:27:51.4329077Z while self.step(): 2024-08-20T22:27:51.4329861Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:27:51.4330676Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:27:51.4331876Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:27:51.4332708Z return inner_fn(self, inst) 2024-08-20T22:27:51.4333607Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:27:51.4334551Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:27:51.4335664Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:27:51.4336584Z return handler(tx, args, kwargs) 2024-08-20T22:27:51.4337644Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:27:51.4338613Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:27:51.4339600Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:27:51.4340604Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:27:51.4341758Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:27:51.4342846Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:27:51.4344042Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:27:51.4344804Z raise UserError( # noqa: B904 2024-08-20T22:27:51.4347909Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:27:51.4350245Z 2024-08-20T22:27:51.4350534Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:27:51.4351582Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:27:51.4353353Z return node.target(*args, **kwargs) 2024-08-20T22:27:51.4353659Z 2024-08-20T22:27:51.4353854Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:27:51.4354651Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:27:51.4355630Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4356830Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:27:51.4357671Z 2024-08-20T22:27:51.4357810Z User Stack (most recent call last): 2024-08-20T22:27:51.4358236Z (snipped, see stack below for prefix) 2024-08-20T22:27:51.4359299Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:51.4360123Z return self.inference(batched_inputs) 2024-08-20T22:27:51.4361057Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:51.4361928Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:51.4362949Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:51.4363824Z images = ImageList.from_tensors( 2024-08-20T22:27:51.4364923Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:51.4365996Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:51.4366410Z 2024-08-20T22:27:51.4366654Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:27:51.4367813Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:27:51.4368611Z 2024-08-20T22:27:51.4368726Z from user code: 2024-08-20T22:27:51.4369569Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:27:51.4370447Z return self.inference(batched_inputs) 2024-08-20T22:27:51.4371701Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:27:51.4372659Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:27:51.4373757Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:27:51.4374692Z images = ImageList.from_tensors( 2024-08-20T22:27:51.4375700Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:27:51.4376805Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:27:51.4377227Z 2024-08-20T22:27:51.4377516Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:27:51.4377982Z 2024-08-20T22:27:51.4378273Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:27:51.4378830Z fail_to_run 2024-08-20T22:27:51.4379439Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:27:51.4380131Z TIMING: 2024-08-20T22:27:51.4380839Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:48 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:27:51.4381731Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:27:54.4857975Z 2024-08-20T22:28:02.8024215Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:28:02.8024868Z loading model: 0it [00:08, ?it/s] 2024-08-20T22:28:02.8025570Z cpu eval detectron2_maskrcnn_r_50_fpn 2024-08-20T22:28:17.0888161Z WARNING:common:fp64 golden ref were not generated for detectron2_maskrcnn_r_50_fpn. Setting accuracy check to cosine 2024-08-20T22:28:17.1304862Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:28.9392706Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:40.6701406Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:46.9397610Z W0820 22:28:46.938810 18451 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Eq(u0, 800), hint=None, expect_rational=True, size_oblivious=True, forcing_spec=False 2024-08-20T22:28:46.9399962Z E0820 22:28:46.938968 18451 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Eq(u0, 800), None), **{'fx_node': eq_3, 'size_oblivious': True}) 2024-08-20T22:28:46.9663266Z ERROR:common: 2024-08-20T22:28:46.9663626Z Traceback (most recent call last): 2024-08-20T22:28:46.9664590Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:28:46.9669177Z return node.target(*args, **kwargs) 2024-08-20T22:28:46.9670543Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:28:46.9671706Z r = self.shape_env.evaluate_expr( 2024-08-20T22:28:46.9673138Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:28:46.9674042Z return retlog(event.run(self)) 2024-08-20T22:28:46.9675375Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:28:46.9676599Z return self.f(*args, **kwargs) 2024-08-20T22:28:46.9678091Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:28:46.9679886Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:28:46.9681336Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:28:46.9682308Z raise self._make_data_dependent_error( 2024-08-20T22:28:46.9683642Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:28:46.9684668Z 2024-08-20T22:28:46.9684950Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:28:46.9685974Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:28:46.9686756Z return node.target(*args, **kwargs) 2024-08-20T22:28:46.9687056Z 2024-08-20T22:28:46.9687250Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:28:46.9688040Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:28:46.9689016Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9690223Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:28:46.9691163Z 2024-08-20T22:28:46.9691302Z User Stack (most recent call last): 2024-08-20T22:28:46.9691729Z (snipped, see stack below for prefix) 2024-08-20T22:28:46.9692946Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:28:46.9693833Z return self.inference(batched_inputs) 2024-08-20T22:28:46.9694853Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:28:46.9695793Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:28:46.9696886Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:28:46.9697936Z images = ImageList.from_tensors( 2024-08-20T22:28:46.9698920Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:28:46.9700012Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:28:46.9700420Z 2024-08-20T22:28:46.9700685Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9701090Z 2024-08-20T22:28:46.9701393Z The above exception was the direct cause of the following exception: 2024-08-20T22:28:46.9701839Z 2024-08-20T22:28:46.9701973Z Traceback (most recent call last): 2024-08-20T22:28:46.9702901Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1999, in get_fake_value 2024-08-20T22:28:46.9703722Z ret_val = wrap_fake_exception( 2024-08-20T22:28:46.9704649Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 1484, in wrap_fake_exception 2024-08-20T22:28:46.9705460Z return fn() 2024-08-20T22:28:46.9706228Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2000, in 2024-08-20T22:28:46.9707110Z lambda: run_node(tx.output, node, args, kwargs, nnmodule) 2024-08-20T22:28:46.9708100Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2135, in run_node 2024-08-20T22:28:46.9708985Z raise RuntimeError(make_error_message(e)).with_traceback( 2024-08-20T22:28:46.9709947Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:28:46.9710741Z return node.target(*args, **kwargs) 2024-08-20T22:28:46.9711786Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 503, in guard_size_oblivious 2024-08-20T22:28:46.9712723Z r = self.shape_env.evaluate_expr( 2024-08-20T22:28:46.9713680Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:28:46.9714548Z return retlog(event.run(self)) 2024-08-20T22:28:46.9715470Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:28:46.9716301Z return self.f(*args, **kwargs) 2024-08-20T22:28:46.9717323Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:28:46.9718635Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:28:46.9720074Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5247, in _evaluate_expr 2024-08-20T22:28:46.9721042Z raise self._make_data_dependent_error( 2024-08-20T22:28:46.9722303Z RuntimeError: Failed running call_function (*(FakeTensor(..., size=(4, 3, u0, u1)), (0, Ellipsis, slice(None, 800, None), slice(None, 1199, None))), **{}): 2024-08-20T22:28:46.9723824Z Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:28:46.9724465Z 2024-08-20T22:28:46.9724734Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:28:46.9725835Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:28:46.9726636Z return node.target(*args, **kwargs) 2024-08-20T22:28:46.9726922Z 2024-08-20T22:28:46.9727112Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:28:46.9727903Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:28:46.9728875Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9730093Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:28:46.9731082Z 2024-08-20T22:28:46.9731231Z User Stack (most recent call last): 2024-08-20T22:28:46.9731645Z (snipped, see stack below for prefix) 2024-08-20T22:28:46.9732654Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:28:46.9733551Z return self.inference(batched_inputs) 2024-08-20T22:28:46.9734564Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:28:46.9735482Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:28:46.9736580Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:28:46.9737507Z images = ImageList.from_tensors( 2024-08-20T22:28:46.9738498Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:28:46.9739568Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:28:46.9739996Z 2024-08-20T22:28:46.9740244Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9740657Z 2024-08-20T22:28:46.9740941Z During handling of the above exception, another exception occurred: 2024-08-20T22:28:46.9741385Z 2024-08-20T22:28:46.9741531Z Traceback (most recent call last): 2024-08-20T22:28:46.9742253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:28:46.9743006Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:28:46.9743983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:28:46.9744865Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:28:46.9746051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:28:46.9746769Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:28:46.9747797Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:28:46.9748687Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:28:46.9749615Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:28:46.9750412Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:28:46.9751432Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:28:46.9752342Z return self._call_impl(*args, **kwargs) 2024-08-20T22:28:46.9753293Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:28:46.9754125Z return forward_call(*args, **kwargs) 2024-08-20T22:28:46.9754999Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:28:46.9755778Z return fn(*args, **kwargs) 2024-08-20T22:28:46.9756729Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:28:46.9757639Z return self._call_impl(*args, **kwargs) 2024-08-20T22:28:46.9758672Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:28:46.9759636Z return forward_call(*args, **kwargs) 2024-08-20T22:28:46.9760566Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:28:46.9761398Z return self._torchdynamo_orig_callable( 2024-08-20T22:28:46.9762312Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:28:46.9763095Z return _compile( 2024-08-20T22:28:46.9763897Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:28:46.9764966Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:28:46.9766098Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:28:46.9766976Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:28:46.9767937Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:28:46.9768717Z return function(*args, **kwargs) 2024-08-20T22:28:46.9769589Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:28:46.9770441Z out_code = transform_code_object(code, transform) 2024-08-20T22:28:46.9771747Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:28:46.9772765Z transformations(instructions, code_options) 2024-08-20T22:28:46.9773718Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:28:46.9774496Z return fn(*args, **kwargs) 2024-08-20T22:28:46.9775379Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:28:46.9776182Z tracer.run() 2024-08-20T22:28:46.9776991Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:28:46.9777775Z super().run() 2024-08-20T22:28:46.9778577Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:28:46.9779370Z while self.step(): 2024-08-20T22:28:46.9780211Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:28:46.9781062Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:28:46.9782054Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:28:46.9782886Z return inner_fn(self, inst) 2024-08-20T22:28:46.9783946Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:28:46.9784861Z self.call_function(fn, args, {}) 2024-08-20T22:28:46.9785765Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:28:46.9786809Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:28:46.9787908Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:28:46.9788759Z return super().call_function(tx, args, kwargs) 2024-08-20T22:28:46.9789731Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:28:46.9790601Z return super().call_function(tx, args, kwargs) 2024-08-20T22:28:46.9791565Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:28:46.9792577Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:28:46.9793837Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:28:46.9794919Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:28:46.9796020Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:28:46.9796896Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:28:46.9797862Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:28:46.9798712Z tracer.run() 2024-08-20T22:28:46.9799463Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:28:46.9800208Z while self.step(): 2024-08-20T22:28:46.9800967Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:28:46.9801774Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:28:46.9802689Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:28:46.9803458Z return inner_fn(self, inst) 2024-08-20T22:28:46.9804324Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:28:46.9805137Z self.call_function(fn, args, {}) 2024-08-20T22:28:46.9806205Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:28:46.9807302Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:28:46.9808454Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:28:46.9809351Z return super().call_function(tx, args, kwargs) 2024-08-20T22:28:46.9810374Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:28:46.9811361Z return super().call_function(tx, args, kwargs) 2024-08-20T22:28:46.9812576Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:28:46.9813655Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:28:46.9814931Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:28:46.9816088Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:28:46.9817272Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:28:46.9818208Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:28:46.9819246Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:28:46.9820093Z tracer.run() 2024-08-20T22:28:46.9820897Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:28:46.9821692Z while self.step(): 2024-08-20T22:28:46.9822518Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:28:46.9823389Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:28:46.9824422Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:28:46.9825195Z return inner_fn(self, inst) 2024-08-20T22:28:46.9826080Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:28:46.9826922Z self.call_function(fn, args, kwargs) 2024-08-20T22:28:46.9827898Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:28:46.9828948Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:28:46.9830034Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:28:46.9830907Z return super().call_function(tx, args, kwargs) 2024-08-20T22:28:46.9831879Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:28:46.9832956Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:28:46.9834142Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:28:46.9835199Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:28:46.9836299Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:28:46.9837173Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:28:46.9838145Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:28:46.9838917Z tracer.run() 2024-08-20T22:28:46.9839664Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:28:46.9840405Z while self.step(): 2024-08-20T22:28:46.9841184Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:28:46.9841977Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:28:46.9842885Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:28:46.9843645Z return inner_fn(self, inst) 2024-08-20T22:28:46.9844466Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 297, in impl 2024-08-20T22:28:46.9845486Z self.push(fn_var.call_function(self, self.popn(nargs), {})) 2024-08-20T22:28:46.9846767Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 982, in call_function 2024-08-20T22:28:46.9847647Z return handler(tx, args, kwargs) 2024-08-20T22:28:46.9848725Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builtin.py", line 958, in _handle_insert_op_in_graph 2024-08-20T22:28:46.9849693Z return wrap_fx_proxy(tx, proxy) 2024-08-20T22:28:46.9850654Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 1946, in wrap_fx_proxy 2024-08-20T22:28:46.9851723Z return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs) 2024-08-20T22:28:46.9852882Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/builder.py", line 2033, in wrap_fx_proxy_cls 2024-08-20T22:28:46.9853979Z example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True) 2024-08-20T22:28:46.9855066Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2054, in get_fake_value 2024-08-20T22:28:46.9855892Z raise UserError( # noqa: B904 2024-08-20T22:28:46.9858604Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time. You will need to explicitly give hint to the compiler. Please take a look at torch._check OR torch._check_is_size APIs. Could not guard on data-dependent expression Eq(u0, 800) (unhinted: Eq(u0, 800)). (Size-like symbols: u0) 2024-08-20T22:28:46.9860894Z 2024-08-20T22:28:46.9861167Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:28:46.9862204Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 2117, in run_node 2024-08-20T22:28:46.9863257Z return node.target(*args, **kwargs) 2024-08-20T22:28:46.9863532Z 2024-08-20T22:28:46.9863709Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:28:46.9864450Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:28:46.9865358Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9866479Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:28:46.9867314Z 2024-08-20T22:28:46.9867450Z User Stack (most recent call last): 2024-08-20T22:28:46.9867841Z (snipped, see stack below for prefix) 2024-08-20T22:28:46.9868778Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:28:46.9869612Z return self.inference(batched_inputs) 2024-08-20T22:28:46.9870548Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:28:46.9871404Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:28:46.9872423Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:28:46.9873476Z images = ImageList.from_tensors( 2024-08-20T22:28:46.9874445Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:28:46.9875494Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:28:46.9875910Z 2024-08-20T22:28:46.9876150Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:28:46.9877298Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:28:46.9878089Z 2024-08-20T22:28:46.9878203Z from user code: 2024-08-20T22:28:46.9879067Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-20T22:28:46.9879928Z return self.inference(batched_inputs) 2024-08-20T22:28:46.9880912Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 203, in inference 2024-08-20T22:28:46.9881822Z images = self.preprocess_image(batched_inputs) 2024-08-20T22:28:46.9882895Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 229, in preprocess_image 2024-08-20T22:28:46.9883793Z images = ImageList.from_tensors( 2024-08-20T22:28:46.9884764Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 127, in from_tensors 2024-08-20T22:28:46.9885883Z batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img) 2024-08-20T22:28:46.9886266Z 2024-08-20T22:28:46.9886550Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:28:46.9886963Z 2024-08-20T22:28:46.9887233Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:28:46.9887752Z fail_to_run 2024-08-20T22:28:46.9888311Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:46.9888963Z TIMING: 2024-08-20T22:28:46.9889474Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:51 | FakeTensor.__torch_dispatch__:1 2024-08-20T22:28:46.9890298Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:28:50.0918249Z 2024-08-20T22:28:58.3885304Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:28:58.3886072Z loading model: 0it [00:08, ?it/s] 2024-08-20T22:28:58.3886744Z cpu eval dlrm 2024-08-20T22:28:59.0591196Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:59.2838394Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:28:59.5077917Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:25.2574070Z Error: weights_offset must be aligned to 16K boundary 2024-08-20T22:29:25.2579139Z ERROR:common: 2024-08-20T22:29:25.2579466Z Traceback (most recent call last): 2024-08-20T22:29:25.2580773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:29:25.2581557Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:29:25.2584094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:29:25.2585101Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:29:25.2585961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1380, in load 2024-08-20T22:29:25.2586750Z cls.cache[key] = torch._export.aot_load(so_path, device) 2024-08-20T22:29:25.2587917Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_export/__init__.py", line 300, in aot_load 2024-08-20T22:29:25.2589105Z runner = torch._C._aoti.AOTIModelContainerRunnerCpu(so_path, 1) # type: ignore[call-arg] 2024-08-20T22:29:25.2590869Z RuntimeError: create_func_( &container_handle_, num_models, device_str.c_str(), cubin_dir.empty() ? nullptr : cubin_dir.c_str()) API call failed at /var/lib/jenkins/workspace/torch/csrc/inductor/aoti_runner/model_container_runner.cpp, line 70 2024-08-20T22:29:25.2592526Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:29:25.2593100Z fail_to_run 2024-08-20T22:29:25.2593718Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:29:25.2594761Z TIMING: backend_compile:9e-05 entire_frame_compile:0.29929 total_wall_time:0.29929 2024-08-20T22:29:25.2595962Z STATS: call_* op count: 36 | FakeTensor.__torch_dispatch__:213 | FakeTensorMode.__torch_dispatch__:2671 | ProxyTorchDispatchMode.__torch_dispatch__:662 2024-08-20T22:29:25.2597087Z Dynamo produced 1 graphs covering 36 ops with 0 graph breaks (0 unique) 2024-08-20T22:29:28.7833284Z 2024-08-20T22:29:30.0135581Z loading model: 0it [00:00, ?it/s]Downloading https://doctr-static.mindee.com/models?id=v0.7.0/db_resnet50-79bd7d70.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/db_resnet50-79bd7d70.pt 2024-08-20T22:29:30.3126324Z 2024-08-20T22:29:30.3126333Z 2024-08-20T22:29:30.4127737Z 0% 0/102021912 [00:00 2024-08-20T22:30:08.3127735Z [ 2024-08-20T22:30:08.3128460Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/detection/core.py", line 92, in 2024-08-20T22:30:08.3129385Z cv2.morphologyEx(bmap[..., idx], cv2.MORPH_OPEN, self._opening_kernel) 2024-08-20T22:30:08.3129815Z 2024-08-20T22:30:08.3130085Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:30:08.3130504Z 2024-08-20T22:30:08.3130784Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:30:08.3131564Z fail_to_run 2024-08-20T22:30:08.3132160Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:08.3132867Z TIMING: 2024-08-20T22:30:08.3133455Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:605 | FakeTensorMode.__torch_dispatch__:4086 2024-08-20T22:30:08.3134347Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:30:11.4931863Z 2024-08-20T22:30:14.9047508Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:14.9048112Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:30:14.9048681Z cpu eval doctr_reco_predictor 2024-08-20T22:30:15.1274836Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:15.2034215Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:15.2776320Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:17.0517881Z W0820 22:30:17.050848 19505 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Ne(u0, 123), hint=None, expect_rational=True, size_oblivious=False, forcing_spec=False 2024-08-20T22:30:17.0520340Z E0820 22:30:17.050991 19505 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne}) 2024-08-20T22:30:17.0595656Z ERROR:common: 2024-08-20T22:30:17.0596022Z Traceback (most recent call last): 2024-08-20T22:30:17.0597134Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:30:17.0598040Z return guard_scalar(self.sym_num) 2024-08-20T22:30:17.0599115Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 845, in guard_scalar 2024-08-20T22:30:17.0600266Z return guard_bool(a) 2024-08-20T22:30:17.0601249Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1018, in guard_bool 2024-08-20T22:30:17.0602290Z return a.node.guard_bool("", 0) # NB: uses Python backtrace 2024-08-20T22:30:17.0603409Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/sym_node.py", line 451, in guard_bool 2024-08-20T22:30:17.0604479Z r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) 2024-08-20T22:30:17.0605661Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T22:30:17.0606516Z return retlog(event.run(self)) 2024-08-20T22:30:17.0607430Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T22:30:17.0608263Z return self.f(*args, **kwargs) 2024-08-20T22:30:17.0609301Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T22:30:17.0610612Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T22:30:17.0612173Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5168, in _evaluate_expr 2024-08-20T22:30:17.0613150Z concrete_val = compute_concrete_val() 2024-08-20T22:30:17.0614274Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5149, in compute_concrete_val 2024-08-20T22:30:17.0615265Z return self.size_hint(orig_expr) 2024-08-20T22:30:17.0616278Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4671, in size_hint 2024-08-20T22:30:17.0617290Z raise self._make_data_dependent_error(result_expr, expr) 2024-08-20T22:30:17.0618729Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T22:30:17.0619765Z 2024-08-20T22:30:17.0620045Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:30:17.0621187Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:30:17.0622068Z return guard_scalar(self.sym_num) 2024-08-20T22:30:17.0622359Z 2024-08-20T22:30:17.0622550Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:30:17.0623400Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:30:17.0624373Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:30:17.0625591Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:30:17.0626421Z 2024-08-20T22:30:17.0626569Z User Stack (most recent call last): 2024-08-20T22:30:17.0626991Z (snipped, see stack below for prefix) 2024-08-20T22:30:17.0628005Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:30:17.0628946Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:30:17.0630122Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:30:17.0631249Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:30:17.0632520Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:30:17.0633435Z words = [ 2024-08-20T22:30:17.0634343Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0635510Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0636714Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0637815Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0638298Z 2024-08-20T22:30:17.0638553Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:30:17.0638960Z 2024-08-20T22:30:17.0639262Z During handling of the above exception, another exception occurred: 2024-08-20T22:30:17.0639704Z 2024-08-20T22:30:17.0639849Z Traceback (most recent call last): 2024-08-20T22:30:17.0640550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:30:17.0641310Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:30:17.0642106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:30:17.0643065Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:30:17.0643905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:30:17.0644628Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:30:17.0645804Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:30:17.0646738Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:30:17.0647658Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T22:30:17.0648478Z result_traced = opt_f(*args, **kwargs) 2024-08-20T22:30:17.0649491Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:30:17.0650396Z return self._call_impl(*args, **kwargs) 2024-08-20T22:30:17.0651421Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:30:17.0652277Z return forward_call(*args, **kwargs) 2024-08-20T22:30:17.0653163Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T22:30:17.0653932Z return fn(*args, **kwargs) 2024-08-20T22:30:17.0654886Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:30:17.0655779Z return self._call_impl(*args, **kwargs) 2024-08-20T22:30:17.0656736Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:30:17.0657581Z return forward_call(*args, **kwargs) 2024-08-20T22:30:17.0658515Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T22:30:17.0659350Z return self._torchdynamo_orig_callable( 2024-08-20T22:30:17.0660303Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T22:30:17.0661103Z return _compile( 2024-08-20T22:30:17.0661935Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T22:30:17.0662984Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:30:17.0664158Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T22:30:17.0665079Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T22:30:17.0666092Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T22:30:17.0667000Z return function(*args, **kwargs) 2024-08-20T22:30:17.0667889Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T22:30:17.0668819Z out_code = transform_code_object(code, transform) 2024-08-20T22:30:17.0670054Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T22:30:17.0671043Z transformations(instructions, code_options) 2024-08-20T22:30:17.0672147Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T22:30:17.0672951Z return fn(*args, **kwargs) 2024-08-20T22:30:17.0673844Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T22:30:17.0674713Z tracer.run() 2024-08-20T22:30:17.0675534Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T22:30:17.0676339Z super().run() 2024-08-20T22:30:17.0677143Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:30:17.0677950Z while self.step(): 2024-08-20T22:30:17.0678770Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:30:17.0679628Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:30:17.0680609Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:30:17.0681443Z return inner_fn(self, inst) 2024-08-20T22:30:17.0682392Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:30:17.0683256Z self.call_function(fn, args, {}) 2024-08-20T22:30:17.0684222Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:30:17.0685348Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:30:17.0686547Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/user_defined.py", line 876, in call_function 2024-08-20T22:30:17.0687622Z return self.call_method(tx, "__call__", args, kwargs) 2024-08-20T22:30:17.0688680Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/user_defined.py", line 736, in call_method 2024-08-20T22:30:17.0689726Z return UserMethodVariable(method, self, source=source).call_function( 2024-08-20T22:30:17.0690958Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T22:30:17.0692047Z return super().call_function(tx, args, kwargs) 2024-08-20T22:30:17.0693109Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:30:17.0694047Z return super().call_function(tx, args, kwargs) 2024-08-20T22:30:17.0695102Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:30:17.0696205Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:30:17.0697468Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:30:17.0698619Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:30:17.0699877Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:30:17.0700828Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:30:17.0701868Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:30:17.0702718Z tracer.run() 2024-08-20T22:30:17.0703635Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:30:17.0704414Z while self.step(): 2024-08-20T22:30:17.0705304Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:30:17.0706151Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:30:17.0707116Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:30:17.0708021Z return inner_fn(self, inst) 2024-08-20T22:30:17.0708915Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T22:30:17.0709753Z self.call_function(fn, args, kwargs) 2024-08-20T22:30:17.0710661Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:30:17.0711697Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:30:17.0712790Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T22:30:17.0713651Z return super().call_function(tx, args, kwargs) 2024-08-20T22:30:17.0714623Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:30:17.0715629Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:30:17.0716810Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:30:17.0717865Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:30:17.0718955Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:30:17.0719820Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:30:17.0720788Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:30:17.0721771Z tracer.run() 2024-08-20T22:30:17.0722547Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:30:17.0723327Z while self.step(): 2024-08-20T22:30:17.0724144Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:30:17.0724991Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:30:17.0725937Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:30:17.0726741Z return inner_fn(self, inst) 2024-08-20T22:30:17.0727662Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:30:17.0728700Z self.call_function(fn, args, {}) 2024-08-20T22:30:17.0729654Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:30:17.0730867Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:30:17.0732065Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:30:17.0733163Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:30:17.0734507Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:30:17.0735674Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:30:17.0736862Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:30:17.0737805Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:30:17.0738863Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:30:17.0739758Z tracer.run() 2024-08-20T22:30:17.0740570Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:30:17.0741370Z while self.step(): 2024-08-20T22:30:17.0742210Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:30:17.0743075Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:30:17.0744167Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T22:30:17.0744977Z return inner_fn(self, inst) 2024-08-20T22:30:17.0746257Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T22:30:17.0747128Z self.call_function(fn, args, {}) 2024-08-20T22:30:17.0748094Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T22:30:17.0749224Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T22:30:17.0750424Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T22:30:17.0751511Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T22:30:17.0752778Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T22:30:17.0753936Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T22:30:17.0755117Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T22:30:17.0756056Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T22:30:17.0757092Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T22:30:17.0757955Z tracer.run() 2024-08-20T22:30:17.0758861Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T22:30:17.0759643Z while self.step(): 2024-08-20T22:30:17.0760450Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T22:30:17.0761299Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T22:30:17.0762244Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 532, in inner 2024-08-20T22:30:17.0763099Z eval_result = value.evaluate_expr(self.output) 2024-08-20T22:30:17.0764096Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1105, in evaluate_expr 2024-08-20T22:30:17.0764953Z raise UserError( # noqa: B904 2024-08-20T22:30:17.0766271Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T22:30:17.0767313Z 2024-08-20T22:30:17.0767589Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T22:30:17.0768774Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T22:30:17.0769681Z return guard_scalar(self.sym_num) 2024-08-20T22:30:17.0769958Z 2024-08-20T22:30:17.0770136Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T22:30:17.0770943Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T22:30:17.0772072Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:30:17.0773279Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T22:30:17.0774198Z 2024-08-20T22:30:17.0774333Z User Stack (most recent call last): 2024-08-20T22:30:17.0774762Z (snipped, see stack below for prefix) 2024-08-20T22:30:17.0775798Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:30:17.0776748Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:30:17.0777790Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:30:17.0778911Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:30:17.0780192Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:30:17.0781102Z words = [ 2024-08-20T22:30:17.0781994Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0783108Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0784412Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0785482Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0785935Z 2024-08-20T22:30:17.0786188Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T22:30:17.0787332Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T22:30:17.0788135Z 2024-08-20T22:30:17.0788240Z from user code: 2024-08-20T22:30:17.0789119Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T22:30:17.0790040Z out["preds"] = self.postprocessor(logits) 2024-08-20T22:30:17.0791046Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T22:30:17.0792144Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T22:30:17.0793385Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T22:30:17.0794264Z words = [ 2024-08-20T22:30:17.0795140Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0796215Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0797391Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T22:30:17.0798551Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T22:30:17.0798980Z 2024-08-20T22:30:17.0799258Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T22:30:17.0799678Z 2024-08-20T22:30:17.0799948Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:30:17.0800457Z fail_to_run 2024-08-20T22:30:17.0801011Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:17.0801662Z TIMING: 2024-08-20T22:30:17.0802243Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:260 | FakeTensorMode.__torch_dispatch__:6321 2024-08-20T22:30:17.0803072Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:30:20.2545059Z 2024-08-20T22:30:21.0205465Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:21.0206060Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:21.0206591Z cpu eval drq 2024-08-20T22:30:21.0295567Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:21.0363534Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:21.0424642Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:21.4523999Z ERROR:common: 2024-08-20T22:30:21.4524723Z Traceback (most recent call last): 2024-08-20T22:30:21.4525642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T22:30:21.4526420Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T22:30:21.4527228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T22:30:21.4528190Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T22:30:21.4529053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T22:30:21.4529767Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T22:30:21.4531078Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T22:30:21.4531983Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T22:30:21.4532924Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1574, in inner 2024-08-20T22:30:21.4533707Z graph = rewrite_signature( 2024-08-20T22:30:21.4534647Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1077, in rewrite_signature 2024-08-20T22:30:21.4535715Z check_user_input_output(flat_results_traced, UserErrorType.INVALID_OUTPUT) 2024-08-20T22:30:21.4536923Z File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 1063, in check_user_input_output 2024-08-20T22:30:21.4537802Z raise UserError( 2024-08-20T22:30:21.4539052Z torch._dynamo.exc.UserError: It looks like one of the outputs with type `` is not supported or pytree-flattenable. 2024-08-20T22:30:21.4542763Z Exported graphs outputs can only contain the following supported types: [, , , , , , , , , , , , , , , , , , ]. 2024-08-20T22:30:21.4546334Z If you are using a custom class object, please register a pytree_flatten/unflatten function using `torch.utils._pytree.register_pytree_node` or `torch.export.register_dataclass`. 2024-08-20T22:30:21.4547664Z TorchDynamo optimized model failed to run because of following error 2024-08-20T22:30:21.4548223Z fail_to_run 2024-08-20T22:30:21.4548828Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:21.4549861Z TIMING: backend_compile:8e-05 entire_frame_compile:0.25023 total_wall_time:0.25023 2024-08-20T22:30:21.4550804Z STATS: call_* op count: 29 | FakeTensor.__torch_dispatch__:66 | FakeTensorMode.__torch_dispatch__:349 2024-08-20T22:30:21.4551692Z Dynamo produced 1 graphs covering 29 ops with 0 graph breaks (0 unique) 2024-08-20T22:30:24.4675728Z 2024-08-20T22:30:24.8643465Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:24.8644056Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:24.8644962Z cpu eval functorch_dp_cifar10 2024-08-20T22:30:24.9061828Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:24.9309251Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:24.9542797Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:53.7048554Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp: In function ‘void cpp_fused_add_native_group_norm_relu_8(bfloat16*, const bfloat16*, const bfloat16*, const float*, const float*, const float*, const float*, const float*, const float*, float*, float*)’: 2024-08-20T22:30:53.7056130Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp:1393:26: warning: variable ‘tmp33’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:30:53.7057785Z 1393 | auto tmp33 = at::vec::convert(tmp32); 2024-08-20T22:30:53.7058309Z | ^~~~~ 2024-08-20T22:30:53.7060833Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp: In function ‘void cpp_fused_add_native_group_norm_relu_13(bfloat16*, const bfloat16*, const bfloat16*, const float*, const float*, const float*, const float*, const float*, const float*, float*, float*)’: 2024-08-20T22:30:53.7064127Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp:1865:26: warning: variable ‘tmp33’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:30:53.7065705Z 1865 | auto tmp33 = at::vec::convert(tmp32); 2024-08-20T22:30:53.7066222Z | ^~~~~ 2024-08-20T22:30:53.7068575Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp: In function ‘void cpp_fused_add_native_group_norm_relu_18(bfloat16*, const bfloat16*, const bfloat16*, const float*, const float*, const float*, const float*, const float*, const float*, float*, float*)’: 2024-08-20T22:30:53.7072236Z /tmp/tmpk7nvzfpp/c7vlo2hz3rnc3qhjjqsuc75ymy6xy3dvgncmypsfke2q43fvsxvb/cpb2oargppx3xqqfzn3vqlgwcmzyx5gg3oeavukushylg34zpeqz.cpp:2239:22: warning: variable ‘tmp35’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:30:53.7073795Z 2239 | auto tmp35 = at::vec::convert(tmp34); 2024-08-20T22:30:53.7074310Z | ^~~~~ 2024-08-20T22:30:54.5093904Z pass 2024-08-20T22:30:54.5096215Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:54.5097816Z TIMING: backend_compile:9e-05 entire_frame_compile:0.42293 total_wall_time:0.42293 2024-08-20T22:30:54.5099442Z STATS: call_* op count: 69 | FakeTensor.__torch_dispatch__:1522 | FakeTensorMode.__torch_dispatch__:6573 | ProxyTorchDispatchMode.__torch_dispatch__:1581 2024-08-20T22:30:54.5100979Z Dynamo produced 1 graphs covering 69 ops with 0 graph breaks (0 unique) 2024-08-20T22:30:57.9454068Z 2024-08-20T22:30:58.0939212Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:58.0939787Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:58.0940353Z cpu eval functorch_maml_omniglot 2024-08-20T22:30:58.0988788Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:58.1038269Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:30:58.1078756Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:19.3796267Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_1(const bfloat16*, const float*, const float*, float*, float*, bfloat16*, bfloat16*)’: 2024-08-20T22:31:19.3804683Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp:494:36: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:31:19.3807692Z 494 | Welford tmp_acc0 = Welford(); 2024-08-20T22:31:19.3808611Z | ^~~~~~~~ 2024-08-20T22:31:19.3812736Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_2(const bfloat16*, const float*, const float*, float*, float*, bfloat16*, bfloat16*)’: 2024-08-20T22:31:19.3818040Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp:585:32: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:31:19.3821030Z 585 | Welford tmp_acc0 = Welford(); 2024-08-20T22:31:19.3821986Z | ^~~~~~~~ 2024-08-20T22:31:19.3826086Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp: In function ‘void cpp_fused__native_batch_norm_legit_max_pool2d_with_indices_relu_3(const bfloat16*, const float*, const float*, float*, float*, bfloat16*, bfloat16*, bfloat16*)’: 2024-08-20T22:31:19.3830644Z /tmp/tmpk7nvzfpp/c5ibghnh7eby2o3hpvfo6vtnfzg3fbkpe3mfac4tk2mvchtuwize/cburqeldgo7l4wsei3piqn6jxx7u22gkxrt22txrw2ah4eet24f2.cpp:677:32: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:31:19.3832653Z 677 | Welford tmp_acc0 = Welford(); 2024-08-20T22:31:19.3833322Z | ^~~~~~~~ 2024-08-20T22:31:19.6449499Z pass 2024-08-20T22:31:19.6452330Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:19.6454391Z TIMING: backend_compile:8e-05 entire_frame_compile:0.18953 total_wall_time:0.18953 2024-08-20T22:31:19.6456470Z STATS: call_* op count: 14 | FakeTensor.__torch_dispatch__:248 | FakeTensorMode.__torch_dispatch__:2140 | ProxyTorchDispatchMode.__torch_dispatch__:400 2024-08-20T22:31:19.6457974Z Dynamo produced 1 graphs covering 14 ops with 0 graph breaks (0 unique) 2024-08-20T22:31:22.9062537Z 2024-08-20T22:31:24.6806316Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:24.6808175Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:31:24.6809274Z cpu eval hf_Albert 2024-08-20T22:31:25.9540532Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:27.4360631Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:31:28.9026990Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:15.1115343Z pass 2024-08-20T22:32:15.1117189Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:15.1118171Z TIMING: backend_compile:0.00014 entire_frame_compile:1.68105 total_wall_time:1.68105 2024-08-20T22:32:15.1119403Z STATS: call_* op count: 436 | FakeTensorMode.__torch_dispatch__:21341 | FakeTensor.__torch_dispatch__:3492 | ProxyTorchDispatchMode.__torch_dispatch__:6914 2024-08-20T22:32:15.1120576Z Dynamo produced 1 graphs covering 436 ops with 0 graph breaks (0 unique) 2024-08-20T22:32:18.7338147Z 2024-08-20T22:32:23.3940937Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:23.3941542Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:32:23.3942105Z cpu eval hf_Bart 2024-08-20T22:32:24.8928762Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:27.0411615Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:32:29.3038867Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:38.6439255Z pass 2024-08-20T22:33:38.6440012Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:38.6443937Z TIMING: backend_compile:0.00016 entire_frame_compile:2.06365 total_wall_time:2.06365 2024-08-20T22:33:38.6447099Z STATS: call_* op count: 451 | FakeTensorMode.__torch_dispatch__:27454 | FakeTensor.__torch_dispatch__:4038 | ProxyTorchDispatchMode.__torch_dispatch__:8066 2024-08-20T22:33:38.6448267Z Dynamo produced 1 graphs covering 451 ops with 0 graph breaks (0 unique) 2024-08-20T22:33:42.5157148Z 2024-08-20T22:33:45.6924414Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:45.6929529Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:33:45.6932101Z cpu eval hf_Bert 2024-08-20T22:33:46.9895315Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:48.7251822Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:33:50.4563839Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:46.1852807Z pass 2024-08-20T22:34:46.1853818Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:34:46.1854816Z TIMING: backend_compile:0.00015 entire_frame_compile:3.27713 total_wall_time:3.27713 2024-08-20T22:34:46.1856051Z STATS: call_* op count: 367 | FakeTensorMode.__torch_dispatch__:23510 | FakeTensor.__torch_dispatch__:3238 | ProxyTorchDispatchMode.__torch_dispatch__:7476 2024-08-20T22:34:46.1857220Z Dynamo produced 1 graphs covering 367 ops with 0 graph breaks (0 unique) 2024-08-20T22:34:49.8766292Z 2024-08-20T22:34:56.8015435Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:56.8015929Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:34:56.8016353Z cpu eval hf_Bert_large 2024-08-20T22:34:59.7299430Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:35:04.2200765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:35:08.7147040Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:55.1782269Z pass 2024-08-20T22:36:55.1784880Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:36:55.1785860Z TIMING: backend_compile:0.00024 entire_frame_compile:6.27387 total_wall_time:6.27387 2024-08-20T22:36:55.1787116Z STATS: call_* op count: 715 | FakeTensorMode.__torch_dispatch__:45650 | FakeTensor.__torch_dispatch__:6310 | ProxyTorchDispatchMode.__torch_dispatch__:14592 2024-08-20T22:36:55.1788284Z Dynamo produced 1 graphs covering 715 ops with 0 graph breaks (0 unique) 2024-08-20T22:36:59.2791049Z 2024-08-20T22:37:01.4770870Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:01.4772134Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:37:01.4780666Z cpu eval hf_DistilBert 2024-08-20T22:37:02.4332003Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:03.4353332Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:04.4443312Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:46.2350540Z pass 2024-08-20T22:37:46.2351295Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:46.2352582Z TIMING: backend_compile:0.00011 entire_frame_compile:0.99807 total_wall_time:0.99807 2024-08-20T22:37:46.2353902Z STATS: call_* op count: 203 | FakeTensor.__torch_dispatch__:2009 | FakeTensorMode.__torch_dispatch__:13155 | ProxyTorchDispatchMode.__torch_dispatch__:3879 2024-08-20T22:37:46.2355061Z Dynamo produced 1 graphs covering 203 ops with 0 graph breaks (0 unique) 2024-08-20T22:37:49.7803859Z 2024-08-20T22:37:52.8220185Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:52.8221037Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:37:52.8226354Z cpu eval hf_GPT2 2024-08-20T22:37:53.5674523Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:54.1578360Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:37:54.7276238Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:50.4132964Z pass 2024-08-20T22:38:50.4135791Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:38:50.4137059Z TIMING: backend_compile:0.00024 entire_frame_compile:2.63071 total_wall_time:2.63071 2024-08-20T22:38:50.4139806Z STATS: call_* op count: 622 | FakeTensorMode.__torch_dispatch__:21714 | FakeTensor.__torch_dispatch__:2533 | ProxyTorchDispatchMode.__torch_dispatch__:6581 2024-08-20T22:38:50.4141017Z Dynamo produced 1 graphs covering 622 ops with 0 graph breaks (0 unique) 2024-08-20T22:38:54.1113855Z 2024-08-20T22:39:20.9628182Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:20.9628689Z loading model: 0it [00:26, ?it/s] 2024-08-20T22:39:20.9629103Z cpu eval hf_GPT2_large 2024-08-20T22:39:20.9629546Z pass_due_to_skip 2024-08-20T22:39:20.9630531Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:39:20.9631257Z TIMING: 2024-08-20T22:39:20.9632504Z STATS: call_* op count: 0 2024-08-20T22:39:20.9633109Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:39:24.4454177Z 2024-08-20T22:39:55.9884731Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:55.9885468Z loading model: 0it [00:31, ?it/s] 2024-08-20T22:39:55.9894702Z cpu eval hf_T5_base 2024-08-20T22:40:48.5428292Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:41:20.6103760Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:42:05.4815880Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:44:46.5198941Z pass 2024-08-20T22:44:46.5199712Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:44:46.5201980Z TIMING: backend_compile:0.00041 entire_frame_compile:6.93184 total_wall_time:6.93184 2024-08-20T22:44:46.5203263Z STATS: call_* op count: 1515 | FakeTensorMode.__torch_dispatch__:57335 | FakeTensor.__torch_dispatch__:8532 | ProxyTorchDispatchMode.__torch_dispatch__:17886 2024-08-20T22:44:46.5204427Z Dynamo produced 1 graphs covering 1515 ops with 0 graph breaks (0 unique) 2024-08-20T22:44:51.4168987Z 2024-08-20T22:45:09.3092027Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:09.3092516Z loading model: 0it [00:17, ?it/s] 2024-08-20T22:45:09.3092944Z cpu eval hf_T5_large 2024-08-20T22:45:09.3094056Z pass_due_to_skip 2024-08-20T22:45:09.3096971Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:45:09.3097738Z TIMING: 2024-08-20T22:45:09.3098007Z STATS: call_* op count: 0 2024-08-20T22:45:09.3098521Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:45:12.8031584Z 2024-08-20T22:45:23.8093181Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:23.8094536Z loading model: 0it [00:11, ?it/s] 2024-08-20T22:45:23.8097499Z cpu eval hf_distil_whisper 2024-08-20T22:45:32.9075659Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:45:45.7324278Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:45:58.5323212Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:17.0449597Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp: In function ‘void cpp_fused__to_copy_add_native_layer_norm_2(const bfloat16*, const float*, const float*, const float*, float*, float*, bfloat16*)’: 2024-08-20T22:47:17.0453095Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:532:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0454714Z 532 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0455262Z | ^~~~~~~~ 2024-08-20T22:47:17.0457047Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:571:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0458679Z 571 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0459159Z | ^~~~~~~~ 2024-08-20T22:47:17.0463318Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp: In function ‘void cpp_fused__to_copy_add_native_layer_norm_4(const bfloat16*, const float*, const bfloat16*, const float*, const float*, float*, float*, float*, bfloat16*)’: 2024-08-20T22:47:17.0466381Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:801:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0467902Z 801 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0468530Z | ^~~~~~~~ 2024-08-20T22:47:17.0470070Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:846:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0471506Z 846 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0472008Z | ^~~~~~~~ 2024-08-20T22:47:17.0474114Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp: In function ‘void cpp_fused__to_copy_add_native_layer_norm_5(const bfloat16*, const float*, const bfloat16*, const bfloat16*, const float*, const float*, float*, float*, float*, bfloat16*)’: 2024-08-20T22:47:17.0477074Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:1029:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0478503Z 1029 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0479002Z | ^~~~~~~~ 2024-08-20T22:47:17.0480559Z /tmp/tmpk7nvzfpp/csa4vpe7pkpl5dps3z7z5qrxsruthg4ro6yqd7a6finnv54xrllr/ccqhmpbjxkt2huyvrmscbj5qvvzmf4v2sxthjbi7eq5tkwamcu7d.cpp:1080:40: warning: variable ‘tmp_acc0’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:47:17.0482005Z 1080 | Welford tmp_acc0 = Welford(); 2024-08-20T22:47:17.0482504Z | ^~~~~~~~ 2024-08-20T22:47:48.0313654Z pass 2024-08-20T22:47:48.0314710Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:48.0315763Z TIMING: backend_compile:0.00019 entire_frame_compile:2.76168 total_wall_time:2.76168 2024-08-20T22:47:48.0317002Z STATS: call_* op count: 635 | FakeTensor.__torch_dispatch__:6257 | FakeTensorMode.__torch_dispatch__:42150 | ProxyTorchDispatchMode.__torch_dispatch__:11806 2024-08-20T22:47:48.0318168Z Dynamo produced 1 graphs covering 635 ops with 0 graph breaks (0 unique) 2024-08-20T22:47:52.2978619Z 2024-08-20T22:47:52.4111696Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:52.4112199Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:52.4113079Z cpu eval lennard_jones 2024-08-20T22:47:52.4156073Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:52.4179700Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:47:52.4198739Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:48:13.1448196Z pass 2024-08-20T22:48:13.1450786Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2024-08-20T22:48:13.1452465Z TIMING: backend_compile:8e-05 entire_frame_compile:0.12986 total_wall_time:0.12986 2024-08-20T22:48:13.1454050Z STATS: call_* op count: 9 | FakeTensor.__torch_dispatch__:91 | FakeTensorMode.__torch_dispatch__:1106 | ProxyTorchDispatchMode.__torch_dispatch__:299 2024-08-20T22:48:13.1455180Z Dynamo produced 1 graphs covering 9 ops with 0 graph breaks (0 unique) 2024-08-20T22:48:14.0549244Z accuracy pass_rate=53.85% 2024-08-20T22:48:14.0555273Z calls_captured gmean=0.00x mean=163.513x 2024-08-20T22:48:14.0559529Z unique_graphs gmean=0.00x mean=0.590x 2024-08-20T22:48:14.0563503Z graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:48:14.0567134Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:48:14.0570894Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T22:48:14.0575223Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T22:48:14.0579307Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T22:48:15.0617744Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_aot_inductor_torchbench_amp_freezing_inference.csv 2024-08-20T22:48:15.4015777Z BERT_pytorch XFAIL 2024-08-20T22:48:15.4018938Z Background_Matting XFAIL 2024-08-20T22:48:15.4024014Z LearningToPaint PASS 2024-08-20T22:48:15.4029225Z Super_SloMo PASS 2024-08-20T22:48:15.4034538Z alexnet PASS 2024-08-20T22:48:15.4039694Z basic_gnn_edgecnn PASS 2024-08-20T22:48:15.4044924Z basic_gnn_gcn PASS 2024-08-20T22:48:15.4050915Z basic_gnn_gin PASS 2024-08-20T22:48:15.4056264Z basic_gnn_sage PASS 2024-08-20T22:48:15.4061589Z dcgan PASS 2024-08-20T22:48:15.4066816Z demucs PASS 2024-08-20T22:48:15.4072054Z densenet121 PASS 2024-08-20T22:48:15.4077313Z detectron2_fasterrcnn_r_101_c4 XFAIL 2024-08-20T22:48:15.4082556Z detectron2_fasterrcnn_r_101_dc5 XFAIL 2024-08-20T22:48:15.4087761Z detectron2_fasterrcnn_r_101_fpn XFAIL 2024-08-20T22:48:15.4093223Z detectron2_fasterrcnn_r_50_c4 XFAIL 2024-08-20T22:48:15.4098324Z detectron2_fasterrcnn_r_50_dc5 XFAIL 2024-08-20T22:48:15.4103542Z detectron2_fasterrcnn_r_50_fpn XFAIL 2024-08-20T22:48:15.4108754Z detectron2_maskrcnn_r_101_c4 XFAIL 2024-08-20T22:48:15.4114005Z detectron2_maskrcnn_r_101_fpn XFAIL 2024-08-20T22:48:15.4119210Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-08-20T22:48:15.4124422Z detectron2_maskrcnn_r_50_fpn XFAIL 2024-08-20T22:48:15.4129621Z dlrm XFAIL 2024-08-20T22:48:15.4135302Z doctr_det_predictor XFAIL 2024-08-20T22:48:15.4140227Z doctr_reco_predictor XFAIL 2024-08-20T22:48:15.4145539Z drq XFAIL 2024-08-20T22:48:15.4150947Z functorch_dp_cifar10 PASS 2024-08-20T22:48:15.4156172Z functorch_maml_omniglot PASS 2024-08-20T22:48:15.4161365Z hf_Albert PASS 2024-08-20T22:48:15.4166608Z hf_Bart PASS 2024-08-20T22:48:15.4171812Z hf_Bert PASS 2024-08-20T22:48:15.4177167Z hf_Bert_large PASS 2024-08-20T22:48:15.4182362Z hf_DistilBert PASS 2024-08-20T22:48:15.4187555Z hf_GPT2 PASS 2024-08-20T22:48:15.4192728Z hf_GPT2_large XFAIL 2024-08-20T22:48:15.4197918Z hf_T5_base PASS 2024-08-20T22:48:15.4203099Z hf_T5_large XFAIL 2024-08-20T22:48:15.4208286Z hf_distil_whisper PASS 2024-08-20T22:48:15.4213693Z lennard_jones PASS 2024-08-20T22:48:15.4605843Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_aot_inductor_torchbench_amp_freezing_inference.csv 2024-08-20T22:48:15.7890445Z BERT_pytorch PASS 2024-08-20T22:48:15.7890962Z Background_Matting PASS 2024-08-20T22:48:15.7893933Z LearningToPaint PASS 2024-08-20T22:48:15.7898763Z Super_SloMo PASS 2024-08-20T22:48:15.7903732Z alexnet PASS 2024-08-20T22:48:15.7908789Z basic_gnn_edgecnn PASS 2024-08-20T22:48:15.7913674Z basic_gnn_gcn PASS 2024-08-20T22:48:15.7918621Z basic_gnn_gin PASS 2024-08-20T22:48:15.7923539Z basic_gnn_sage PASS 2024-08-20T22:48:15.7928479Z dcgan PASS 2024-08-20T22:48:15.7933746Z demucs PASS 2024-08-20T22:48:15.7938691Z densenet121 PASS 2024-08-20T22:48:15.7943635Z detectron2_fasterrcnn_r_101_c4 PASS 2024-08-20T22:48:15.7948922Z detectron2_fasterrcnn_r_101_dc5 PASS 2024-08-20T22:48:15.7953860Z detectron2_fasterrcnn_r_101_fpn PASS 2024-08-20T22:48:15.7958810Z detectron2_fasterrcnn_r_50_c4 PASS 2024-08-20T22:48:15.7963733Z detectron2_fasterrcnn_r_50_dc5 PASS 2024-08-20T22:48:15.7968618Z detectron2_fasterrcnn_r_50_fpn PASS 2024-08-20T22:48:15.7973725Z detectron2_maskrcnn_r_101_c4 PASS 2024-08-20T22:48:15.7978683Z detectron2_maskrcnn_r_101_fpn PASS 2024-08-20T22:48:15.7983589Z detectron2_maskrcnn_r_50_c4 PASS 2024-08-20T22:48:15.7988504Z detectron2_maskrcnn_r_50_fpn PASS 2024-08-20T22:48:15.7993414Z dlrm PASS 2024-08-20T22:48:15.7998370Z doctr_det_predictor PASS 2024-08-20T22:48:15.8003298Z doctr_reco_predictor PASS 2024-08-20T22:48:15.8008187Z drq PASS 2024-08-20T22:48:15.8013323Z functorch_dp_cifar10 PASS 2024-08-20T22:48:15.8018263Z functorch_maml_omniglot PASS 2024-08-20T22:48:15.8023202Z hf_Albert PASS 2024-08-20T22:48:15.8028106Z hf_Bart PASS 2024-08-20T22:48:15.8033017Z hf_Bert PASS 2024-08-20T22:48:15.8037952Z hf_Bert_large PASS 2024-08-20T22:48:15.8042867Z hf_DistilBert PASS 2024-08-20T22:48:15.8048162Z hf_GPT2 PASS 2024-08-20T22:48:15.8053287Z hf_GPT2_large PASS 2024-08-20T22:48:15.8058217Z hf_T5_base PASS 2024-08-20T22:48:15.8063141Z hf_T5_large PASS 2024-08-20T22:48:15.8068092Z hf_distil_whisper PASS 2024-08-20T22:48:15.8072995Z lennard_jones PASS 2024-08-20T22:48:15.8460028Z + cleanup_workspace 2024-08-20T22:48:15.8461148Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:48:15.8462278Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:48:15.8463235Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:48:15.8463873Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:48:15.8464703Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:48:15.8465752Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:48:15.8466463Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:48:16.3249286Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:48:16.3249784Z cat test/**/*_toprint.log || true 2024-08-20T22:48:16.3258890Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.3259386Z env: 2024-08-20T22:48:16.3259648Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.3260238Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.3260886Z ##[endgroup] 2024-08-20T22:48:16.3334004Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:48:16.3360733Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:48:16.3361175Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:48:16.3366745Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.3367229Z env: 2024-08-20T22:48:16.3367493Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.3368073Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.3368724Z MONITOR_SCRIPT_PID: 485083 2024-08-20T22:48:16.3369046Z ##[endgroup] 2024-08-20T22:48:16.3577080Z Prepare all required actions 2024-08-20T22:48:16.3577535Z Getting action download info 2024-08-20T22:48:16.6098400Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:48:16.7828548Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:48:16.7828969Z with: 2024-08-20T22:48:16.7829520Z file-suffix: test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055 2024-08-20T22:48:16.7830201Z s3-bucket: gha-artifacts 2024-08-20T22:48:16.7830508Z env: 2024-08-20T22:48:16.7830732Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.7831281Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.7831888Z ##[endgroup] 2024-08-20T22:48:16.7856376Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:48:16.7856998Z # Remove any previous test jsons if they exist 2024-08-20T22:48:16.7857546Z rm -f test-jsons-*.zip 2024-08-20T22:48:16.7858044Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:48:16.7863740Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.7864315Z env: 2024-08-20T22:48:16.7864557Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.7865096Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.7866006Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055 2024-08-20T22:48:16.7866669Z ##[endgroup] 2024-08-20T22:48:16.8094713Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:48:16.8148249Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:48:16.8154430Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:48:16.8166343Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:48:16.8183022Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:48:16.8209803Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:48:16.8210445Z # Remove any previous test reports if they exist 2024-08-20T22:48:16.8210963Z rm -f test-reports-*.zip 2024-08-20T22:48:16.8211546Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:48:16.8217266Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.8217739Z env: 2024-08-20T22:48:16.8217994Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.8218585Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.8219562Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055 2024-08-20T22:48:16.8220405Z ##[endgroup] 2024-08-20T22:48:16.8403532Z adding: test/test-reports/inference_torchbench.csv (deflated 74%) 2024-08-20T22:48:16.8428484Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:48:16.8429090Z # Remove any previous usage logs if they exist 2024-08-20T22:48:16.8429566Z rm -f logs-*.zip 2024-08-20T22:48:16.8430205Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:48:16.8430980Z # so check to see if the file exists first 2024-08-20T22:48:16.8431537Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:48:16.8432011Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:48:16.8432454Z fi 2024-08-20T22:48:16.8432769Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:48:16.8433275Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:48:16.8433726Z fi 2024-08-20T22:48:16.8438529Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.8438980Z env: 2024-08-20T22:48:16.8439228Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.8439926Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.8440818Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055 2024-08-20T22:48:16.8441477Z ##[endgroup] 2024-08-20T22:48:16.8509934Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:48:16.8575848Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:48:16.8576556Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:48:16.8577090Z rm -f debug-*.zip 2024-08-20T22:48:16.8577453Z if [ -d 'test/debug' ]; then 2024-08-20T22:48:16.8577928Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:48:16.8578385Z fi 2024-08-20T22:48:16.8583367Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:16.8583850Z env: 2024-08-20T22:48:16.8584201Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.8584747Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.8585650Z FILE_SUFFIX: test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055 2024-08-20T22:48:16.8586308Z ##[endgroup] 2024-08-20T22:48:16.8676327Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:48:16.8677533Z with: 2024-08-20T22:48:16.8678250Z s3-bucket: gha-artifacts 2024-08-20T22:48:16.8679440Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:16.8680781Z retention-days: 14 2024-08-20T22:48:16.8681060Z if-no-files-found: warn 2024-08-20T22:48:16.8681374Z path: test-jsons-*.zip 2024-08-20T22:48:16.8681675Z name: artifact 2024-08-20T22:48:16.8681923Z region: us-east-1 2024-08-20T22:48:16.8682187Z env: 2024-08-20T22:48:16.8682427Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:16.8682976Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:16.8683558Z ##[endgroup] 2024-08-20T22:48:17.2213123Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:48:17.2213754Z With the provided path, there will be 1 file uploaded 2024-08-20T22:48:17.2214376Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:17.2257819Z Starting upload of test-jsons-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:17.4097574Z Finished upload of test-jsons-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:17.4302595Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:48:17.4303030Z with: 2024-08-20T22:48:17.4303307Z s3-bucket: gha-artifacts 2024-08-20T22:48:17.4303739Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:17.4304389Z retention-days: 14 2024-08-20T22:48:17.4304815Z if-no-files-found: error 2024-08-20T22:48:17.4305167Z path: test-reports-*.zip 2024-08-20T22:48:17.4305500Z name: artifact 2024-08-20T22:48:17.4305772Z region: us-east-1 2024-08-20T22:48:17.4306054Z env: 2024-08-20T22:48:17.4306326Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:17.4306912Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:17.4307528Z ##[endgroup] 2024-08-20T22:48:17.7617156Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:48:17.7617769Z With the provided path, there will be 1 file uploaded 2024-08-20T22:48:17.7618371Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:17.7656836Z Starting upload of test-reports-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:17.8880361Z Finished upload of test-reports-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:17.9079832Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:48:17.9080226Z with: 2024-08-20T22:48:17.9080488Z s3-bucket: gha-artifacts 2024-08-20T22:48:17.9080886Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:17.9081507Z retention-days: 14 2024-08-20T22:48:17.9081799Z if-no-files-found: ignore 2024-08-20T22:48:17.9082128Z path: logs-*.zip 2024-08-20T22:48:17.9082409Z name: artifact 2024-08-20T22:48:17.9082669Z region: us-east-1 2024-08-20T22:48:17.9082940Z env: 2024-08-20T22:48:17.9083183Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:17.9083740Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:17.9084325Z ##[endgroup] 2024-08-20T22:48:18.2348037Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:48:18.2348654Z With the provided path, there will be 1 file uploaded 2024-08-20T22:48:18.2349272Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:18.2386763Z Starting upload of logs-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:18.3266641Z Finished upload of logs-test-cpu_aot_inductor_torchbench_amp_freezing-1-2-amz2023.linux.12xlarge_29025551055.zip 2024-08-20T22:48:18.3472232Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:48:18.3472807Z with: 2024-08-20T22:48:18.3473058Z s3-bucket: gha-artifacts 2024-08-20T22:48:18.3473454Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:48:18.3473874Z retention-days: 14 2024-08-20T22:48:18.3474175Z if-no-files-found: ignore 2024-08-20T22:48:18.3474500Z path: debug-*.zip 2024-08-20T22:48:18.3474777Z name: artifact 2024-08-20T22:48:18.3475032Z region: us-east-1 2024-08-20T22:48:18.3475290Z env: 2024-08-20T22:48:18.3475528Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:18.3476070Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:18.3476653Z ##[endgroup] 2024-08-20T22:48:18.6693302Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:48:18.6877357Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:48:18.6877817Z # shellcheck disable=SC2156 2024-08-20T22:48:18.6878597Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:48:18.6884622Z shell: /usr/bin/bash -e {0} 2024-08-20T22:48:18.6884939Z env: 2024-08-20T22:48:18.6885203Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:18.6885897Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:18.6886486Z ##[endgroup] 2024-08-20T22:48:18.9034977Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:48:18.9035492Z with: 2024-08-20T22:48:18.9035719Z env: 2024-08-20T22:48:18.9035963Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:18.9036511Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:18.9037447Z ##[endgroup] 2024-08-20T22:48:18.9054958Z ##[group]Run set -eou pipefail 2024-08-20T22:48:18.9055350Z set -eou pipefail 2024-08-20T22:48:18.9055686Z  2024-08-20T22:48:18.9056190Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:48:18.9056820Z for _ in $(seq 1440); do 2024-08-20T22:48:18.9057280Z  # Break if no ssh session exists anymore 2024-08-20T22:48:18.9057763Z  if [ "$(who)" = "" ]; then 2024-08-20T22:48:18.9058152Z  break 2024-08-20T22:48:18.9058434Z  fi 2024-08-20T22:48:18.9058760Z  echo "." 2024-08-20T22:48:18.9059058Z  sleep 5 2024-08-20T22:48:18.9059353Z done 2024-08-20T22:48:18.9064939Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:18.9065396Z env: 2024-08-20T22:48:18.9065626Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:18.9066176Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:18.9066784Z ##[endgroup] 2024-08-20T22:48:18.9088621Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:48:18.9151274Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:48:18.9152041Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:48:18.9152630Z # shellcheck disable=SC2046 2024-08-20T22:48:18.9153081Z docker stop $(docker ps -q) || true 2024-08-20T22:48:18.9153535Z # Prune all of the docker images 2024-08-20T22:48:18.9153975Z docker system prune -af 2024-08-20T22:48:18.9159437Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:18.9159925Z env: 2024-08-20T22:48:18.9160173Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:48:18.9160771Z DOCKER_CONTAINER_ID: b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:18.9161514Z ##[endgroup] 2024-08-20T22:48:29.7921154Z b46344175dba 2024-08-20T22:48:31.3700080Z Deleted Containers: 2024-08-20T22:48:31.3700653Z b46344175dba8fcf3e478c349650be2408c89e1f96710004f753527071d640a4 2024-08-20T22:48:31.3701098Z 2024-08-20T22:48:35.1865556Z Deleted Images: 2024-08-20T22:48:35.1867009Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:48:35.1869246Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11-inductor-benchmarks@sha256:ed18f711327455e269271a17448384275684348a472ba289f00904bd14501fa9 2024-08-20T22:48:35.1870760Z deleted: sha256:822356886a68936cce34917f5ffc60d864918fab7594bb08c26bf1816dd6110d 2024-08-20T22:48:35.1871610Z deleted: sha256:ae6b018f5c327102b48941d39cd8d6fa6151893715290829a76e4e73add3a418 2024-08-20T22:48:35.1872482Z deleted: sha256:79fa7dddbf1ed3af5e8f10972d07ccea774929efe8578a1cc6604a00c93c1e02 2024-08-20T22:48:35.1873362Z deleted: sha256:06b439d80ae875bb9b96d0a23c4619aec5b5fb2f35b5048022ebe6061e19f7e1 2024-08-20T22:48:35.1874219Z deleted: sha256:52d3d63b327ae120e60f76fe27965e31638eb8d556b5e5f56f4773bd940373cf 2024-08-20T22:48:35.1875072Z deleted: sha256:5df173ad6b0f1ba16287832b6e5237513d152e87e627e478fc36ed16159697c0 2024-08-20T22:48:35.1875929Z deleted: sha256:b6c13bf75251ea08468f4040dbba32848f72cd00eb8c3003ecc1e879f73738ae 2024-08-20T22:48:35.1876785Z deleted: sha256:f2f37b6c898fc6a8c74683cc50a156f56abe71f20310c72338bb0ec23c71e8d6 2024-08-20T22:48:35.1877649Z deleted: sha256:55832a6aecdb11a84ddbbd38efcac3b62dd61a2b7832a1080535fc5730e1e336 2024-08-20T22:48:35.1878804Z deleted: sha256:bd86c04ac5f8ba1940bc991dfbae9e17a80293b5cf5fe568c694f46d59863011 2024-08-20T22:48:35.1879700Z deleted: sha256:88a76e0add806e67ed7bfd8ad7f862a8683d087dd3557cfc2b787da5f4c1ae0f 2024-08-20T22:48:35.1880657Z deleted: sha256:37a2749f2e5a556bb0369b3db12c47e4b82b41064f430896babceb49d97d2f9f 2024-08-20T22:48:35.1881458Z deleted: sha256:ac6214a417e93e76048bee154136a3117823ef38824b38f2659c74c7d09b57f8 2024-08-20T22:48:35.1882408Z deleted: sha256:bd60508bf2fcb98670e5ba90896321868f6fb9c5a2c859203e47512542f45b99 2024-08-20T22:48:35.1883244Z deleted: sha256:3fe3c9441ff3190f5cf87c066d3b5ca03aa3bb68071ef86e127c271ebb1ea02e 2024-08-20T22:48:35.1884072Z deleted: sha256:f9b8978a3de27376176f0400dcb17127771f2a8123ff323474e055dbae6eaba5 2024-08-20T22:48:35.1884878Z deleted: sha256:825e6cc758ce8478a096de8d367123d4d0a7f51fbe595867d2f950f714ca908e 2024-08-20T22:48:35.1885709Z deleted: sha256:dde33dfb74ab43131553ba775cd8561874596ba2e3d1b810c7796beba3d4fd38 2024-08-20T22:48:35.1886542Z deleted: sha256:809173faaa7c1289371a0c5a0927a6050de23417cddba0746dea79b10b68dd71 2024-08-20T22:48:35.1887361Z deleted: sha256:f63fee6ae14582933c75117b8d3115041d0177792e510d8dab0baca1b2202771 2024-08-20T22:48:35.1888367Z deleted: sha256:9432a4a5f47ddff6148fd5f8ade43d3a5566c0da2fbcd904daa917dc21c4c73d 2024-08-20T22:48:35.1889423Z deleted: sha256:b9ff51a3d1dc73e8a3ab751fa1a512026a101eaedbea74cfe01a2876b5111502 2024-08-20T22:48:35.1890321Z deleted: sha256:70e4013b779cc78a919225cad0a55d38948b0620ea119eb80dbb377ce8687f7e 2024-08-20T22:48:35.1891170Z deleted: sha256:edcf423609137384d678c9f677cae55933016f3ee76cc2ee47d93aad1346c427 2024-08-20T22:48:35.1892116Z deleted: sha256:bb2f6a3359e809db17e8b77499bf0d94200af9defc470b3adb3e1c28a6f1c6c1 2024-08-20T22:48:35.1892955Z deleted: sha256:eb646c0b0638f760f7c1046a113d3646e57513d19390881f4f6775ec702fa265 2024-08-20T22:48:35.1893802Z deleted: sha256:71c5137ede0421f7f00b1e93ee7bd6287c8ff6445615d0e1d1edddff2240789c 2024-08-20T22:48:35.1894644Z deleted: sha256:d2a20794246965e241eaaa98883ecd02a39901618f06536532bb4d9ed2b80f63 2024-08-20T22:48:35.1895481Z deleted: sha256:a8fc5137968c5840b35ff322f4796356303c4d4f0ed5c86c70218e20045936ac 2024-08-20T22:48:35.1896310Z deleted: sha256:1bd367d8cc65cb8808ddf7665064a0523467c7947ead2e1aed5112e9465e2fa8 2024-08-20T22:48:35.1897174Z deleted: sha256:736b605a3af6c5f2f82b517f0c1c517701a78def165ad99f3fb034198818994d 2024-08-20T22:48:35.1898051Z deleted: sha256:f2766fd6c51e0f9c3fdeed91ff144d049ed99f1a2cdce5c3b36a9f694af46045 2024-08-20T22:48:35.1898909Z deleted: sha256:688dce500613b1366475010344298a9b4eca441474fb871cfc37287a6eefbbf4 2024-08-20T22:48:35.1899733Z deleted: sha256:08f9d176a447289b58f9c70cdc76546487fc2700b1e5a7b79c19eb4f8f7c4608 2024-08-20T22:48:35.1900597Z deleted: sha256:538a2fc18bbc7da30a73c02bc8bb61bd6800a9d501caeaec614a32c41d53438d 2024-08-20T22:48:35.1901464Z deleted: sha256:75d7a20f4c66e80cf12fdf6d68787ee75c33478fd7c3e30c4696702ef9932993 2024-08-20T22:48:35.1902309Z deleted: sha256:5f36a35d97f9c931857392a1a616a5af4da00dfabe321b21f7637761d680b575 2024-08-20T22:48:35.1903154Z deleted: sha256:172cb1246a46e9dde7d50405f7fb31fd83a3a3e6ff40c811cfe100bd034807f6 2024-08-20T22:48:35.1904020Z deleted: sha256:d621ca4d3a5a15f4d74797b18bd0f4c44f63288c18e753fc4281cec69d82ea21 2024-08-20T22:48:35.1904928Z deleted: sha256:8e5eec944591edb185a54d6b495949dcae1da0be3fec8d30c3463e9fcde1a34e 2024-08-20T22:48:35.1905956Z deleted: sha256:656151b8c659bb4dab88293de022523e0e478e654f6922eaf53efb36ffb26e7e 2024-08-20T22:48:35.1906797Z deleted: sha256:9e2c8ec2c392f561f1016a988a64620f377203ed895815f024856cdef934be6b 2024-08-20T22:48:35.1907629Z deleted: sha256:c42f2e17dc37198a178852ecf9d947775b76af99ebdc2aa929af4b03b3896b3b 2024-08-20T22:48:35.1908475Z deleted: sha256:f52f63d8f16c18cde14e5738947c6e08323452e8599f723a6e041ed1ab33565a 2024-08-20T22:48:35.1909327Z deleted: sha256:343dc9f9b3e3c25dc528bbe1f8b4258289e512ff85f4b9c22ea2154c9c9d8143 2024-08-20T22:48:35.1910184Z deleted: sha256:5efa5da92a100a8c36c51a732af9534a8023041baf9d55ecf831c1386467d65a 2024-08-20T22:48:35.1911162Z deleted: sha256:da5cd90133fe64eb015b47ab4515d89305598acefc5c097b8e189da64be2d476 2024-08-20T22:48:35.1912025Z deleted: sha256:691e04884c858a1ea0a97381a7b1036bdfc6f9a48471608f0a5db9da9935555f 2024-08-20T22:48:35.1912877Z deleted: sha256:64100fb171394a71add5752ea3f6c432bcba0ac31aca67d8443c3722864804e1 2024-08-20T22:48:35.1913735Z deleted: sha256:a03197ec2ed41935ef46795a87ffd3d80b526fce1edf28550177f579cfb5df64 2024-08-20T22:48:35.1914657Z deleted: sha256:7bd9c2008289af25e772b1fc7b8e6470b513a5f76ce25743b9ef1bbe43ace5f7 2024-08-20T22:48:35.1915501Z deleted: sha256:0064821cdad60c711279a5964f88bf82c7582c296d21a0d1d0e97c60a47c1fa3 2024-08-20T22:48:35.1916341Z deleted: sha256:5825eb9623523140ad1e3ad683f3a83ab39054adf0b0fb4e472c2a7c3e7545ff 2024-08-20T22:48:35.1917200Z deleted: sha256:ff2d63ec14fa462e2edcd48d8ca8b678d79c9981bda17880818db15b8790e54b 2024-08-20T22:48:35.1918066Z deleted: sha256:79a5278d4bd098095d2f18fcfa476e8d54112dae4ac3d6c4c955e7fd9ba550f4 2024-08-20T22:48:35.1919021Z deleted: sha256:808ef6924cb221e2093f74413950d96ff62707fc13a194d83a0422315fccccdc 2024-08-20T22:48:35.1919955Z deleted: sha256:91aa45c12c4226ea9961e3acd079df4454bf2a55cb1bb9a2918f37bec89947bf 2024-08-20T22:48:35.1920753Z deleted: sha256:1f26a67abbf110b4883c6dd39e035f74248d28916c4a48be34296a7a96d1319b 2024-08-20T22:48:35.1921530Z deleted: sha256:1165de789bc53191b61c2b04dc04cf9d3d6d78f3d704776bd5a276d2caac8d09 2024-08-20T22:48:35.1922346Z deleted: sha256:aff63be2ff8d27d7172fb6dbecff0bea191a85c2431f8de65c0f81d40212587b 2024-08-20T22:48:35.1923160Z deleted: sha256:7c4290bd5a95669ca4d362a8672cb96b80693cc40af13f1ce061e86f835766d1 2024-08-20T22:48:35.1923963Z deleted: sha256:46ad9cfd3aa9b54d845ebc37c2e4c0d5fda90577227a7962049b71c825acdafe 2024-08-20T22:48:35.1924774Z deleted: sha256:aad12f955a8ca5766f6ca5d1467a6d2b81be713ff7224e8ae8eb92b68acf69a2 2024-08-20T22:48:35.1925564Z deleted: sha256:f5a053a7d9bda27126802be2ba52d187c26e934b7cc582850d2aaadd90631a1e 2024-08-20T22:48:35.1926359Z deleted: sha256:cf0f3c2eb852401868a3f959f6176f5a6b42f17f16e950f132f03293c764d733 2024-08-20T22:48:35.1927121Z deleted: sha256:7929c61352934651542ff23122529355349b2153871117dcfbaa4b45020a158d 2024-08-20T22:48:35.1927885Z deleted: sha256:119390db59d077c75caa11c49fee6ab586d061d27de1deeeec8608f330624f21 2024-08-20T22:48:35.1928661Z deleted: sha256:3b7424223daa8c5790ef526174f3780d9b1b54baa45d080127bfc72e60139a2b 2024-08-20T22:48:35.1929447Z deleted: sha256:f8986b3dfad5395410b72709f00789a62ee60d1b0bc1b27583f4c17af34b43fc 2024-08-20T22:48:35.1930225Z deleted: sha256:03db7655e84b428b1399087a9ed34180c120808cd00d94924a46eeaef73c37b0 2024-08-20T22:48:35.1931008Z deleted: sha256:b4a81bb8166fcf8ffbf063817b3202900c20586ca4bf369b21822764ab966164 2024-08-20T22:48:35.1931794Z deleted: sha256:afc241c47ba97b4a4f054cd0e6a5d90428b2675d00b160727187b7feabb81b14 2024-08-20T22:48:35.1932876Z deleted: sha256:e04730e6f5fc53e139356efef9c2c84b7bf6087c5a89876deaf1be58b9b1029b 2024-08-20T22:48:35.1933771Z deleted: sha256:7b22c8a2bc4bb0051ddc3ea51c917e7562663e4dafe4f9ad4a3870f4ec84bd14 2024-08-20T22:48:35.1934635Z deleted: sha256:c54b28287358751c9e19bd4f52b1d920ec5f4a254f8883b9d093528c08bc8b3f 2024-08-20T22:48:35.1935477Z deleted: sha256:17303ef9705fd22b570529dc8413ccba1a59fab7f5fef103bfe72e01c53d976e 2024-08-20T22:48:35.1936340Z deleted: sha256:1a4e7403586e712f2caf32f8bec553dafd88f78612fba5bd0c54e4efb89b9c41 2024-08-20T22:48:35.1937211Z deleted: sha256:d2f2462c603417aa43b82ea6f370ed4319cd12b3bda370d5a7ada0845daf3aaa 2024-08-20T22:48:35.1938081Z deleted: sha256:973bcd7518ef8f8e81b3b895b1ff4fe6e8a20915387d1f127edefccdcc3d867d 2024-08-20T22:48:35.1938913Z deleted: sha256:ab913355e8147522ebc4f29618812090844aa39956668afc4b91926827c327b3 2024-08-20T22:48:35.1939742Z deleted: sha256:69527fc1954cb8778833d15c0f3562fda1cbd230b11e649a974752f7a726cbf9 2024-08-20T22:48:35.1940570Z deleted: sha256:1b2a52846c812d7caf91918531cb04230500b31c3f695e444c1885aad0777b71 2024-08-20T22:48:35.1941422Z deleted: sha256:40f793cdc6b0a26ec9af5ea7188ad0ef19799a234e53eb4b8f3f756694a1000e 2024-08-20T22:48:35.1942287Z deleted: sha256:39b098ef95425e2d1bdb44559eda25dfc7bd62af2fb6bb691b540741278acb95 2024-08-20T22:48:35.1943224Z deleted: sha256:bc116c6279a58082c4ef9dac27e9d4d76bfdd78c2f2fe4439663d3dd2af55b57 2024-08-20T22:48:35.1944083Z deleted: sha256:57046a92505d1125e1af1100aaeb0e36175039b7dd36e67e7b4c554997d81b07 2024-08-20T22:48:35.1944913Z deleted: sha256:e868a8284f40af2175a17211327660844c1ac66ee31919f52c5e4e2b9564213a 2024-08-20T22:48:35.1946071Z deleted: sha256:ba5f7b6add2ae68d2962f19c3d981adc39633683af6dc3392c354cce87b7e8f9 2024-08-20T22:48:35.1946913Z deleted: sha256:3a06302b80c87d1030f701f23861d185613a3a4269faf533bb1b82b092a82887 2024-08-20T22:48:35.1947762Z deleted: sha256:dbdee317045b3156074f52d1fb32d613a48b6d8ed13c3e400a9f2c7bcceaf808 2024-08-20T22:48:35.1948628Z deleted: sha256:9574f911b87c39bbbb08a0238b82fb53abdedad1f387beb999ce1d36f3845e98 2024-08-20T22:48:35.1949505Z deleted: sha256:15d49efcad797dff0aa23e3b025b50c1c961668637e4bd9dff2361eaf3405c74 2024-08-20T22:48:35.1950371Z deleted: sha256:1b9b7346fee7abbc7f5538eaa23548bd05a45abe8daf6794024be0c8ad7d60bb 2024-08-20T22:48:35.1950907Z 2024-08-20T22:48:35.1951050Z Total reclaimed space: 35.51GB 2024-08-20T22:48:35.2015756Z Post job cleanup. 2024-08-20T22:48:35.2064318Z Post job cleanup. 2024-08-20T22:48:35.2921728Z [command]/usr/bin/git version 2024-08-20T22:48:35.2978171Z git version 2.40.1 2024-08-20T22:48:35.3016397Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7eafaa56-ab8e-480b-af32-7cb9583b3fa5' before making global git config changes 2024-08-20T22:48:35.3017724Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:48:35.3020711Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:48:35.3056817Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:48:35.3087900Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:48:35.3407720Z Entering 'android/libs/fbjni' 2024-08-20T22:48:35.3468420Z Entering 'third_party/FP16' 2024-08-20T22:48:35.3523444Z Entering 'third_party/FXdiv' 2024-08-20T22:48:35.3573865Z Entering 'third_party/NNPACK' 2024-08-20T22:48:35.3623586Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:48:35.3677129Z Entering 'third_party/XNNPACK' 2024-08-20T22:48:35.3743127Z Entering 'third_party/benchmark' 2024-08-20T22:48:35.3795326Z Entering 'third_party/cpp-httplib' 2024-08-20T22:48:35.3846492Z Entering 'third_party/cpuinfo' 2024-08-20T22:48:35.3896274Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:48:35.3948043Z Entering 'third_party/cutlass' 2024-08-20T22:48:35.4006963Z Entering 'third_party/eigen' 2024-08-20T22:48:35.4058327Z Entering 'third_party/fbgemm' 2024-08-20T22:48:35.4109538Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:48:35.4157154Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:48:35.4207327Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:48:35.4263427Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:48:35.4313669Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:48:35.4364532Z Entering 'third_party/flatbuffers' 2024-08-20T22:48:35.4417466Z Entering 'third_party/fmt' 2024-08-20T22:48:35.4468677Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:48:35.4519155Z Entering 'third_party/gloo' 2024-08-20T22:48:35.4569424Z Entering 'third_party/googletest' 2024-08-20T22:48:35.4618106Z Entering 'third_party/ideep' 2024-08-20T22:48:35.4668755Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:48:35.4724882Z Entering 'third_party/ittapi' 2024-08-20T22:48:35.4776496Z Entering 'third_party/kineto' 2024-08-20T22:48:35.4827798Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:48:35.4877082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:48:35.4927384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:48:35.4981782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:48:35.5031566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:48:35.5084257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:48:35.5134404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:48:35.5186244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:48:35.5237098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:48:35.5288291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:48:35.5338872Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:48:35.5389976Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:48:35.5440659Z Entering 'third_party/mimalloc' 2024-08-20T22:48:35.5494704Z Entering 'third_party/nccl/nccl' 2024-08-20T22:48:35.5544021Z Entering 'third_party/nlohmann' 2024-08-20T22:48:35.5598846Z Entering 'third_party/onnx' 2024-08-20T22:48:35.5665594Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:48:35.5715119Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:48:35.5769803Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:48:35.5820643Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:48:35.5870537Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:48:35.5919343Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:48:35.5968538Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:48:35.6018481Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:48:35.6070378Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:48:35.6119365Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:48:35.6168187Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:48:35.6225800Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:48:35.6277012Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:48:35.6352319Z Entering 'third_party/pocketfft' 2024-08-20T22:48:35.6404433Z Entering 'third_party/protobuf' 2024-08-20T22:48:35.6457399Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:48:35.6514168Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:48:35.6567231Z Entering 'third_party/psimd' 2024-08-20T22:48:35.6621945Z Entering 'third_party/pthreadpool' 2024-08-20T22:48:35.6673054Z Entering 'third_party/pybind11' 2024-08-20T22:48:35.6728773Z Entering 'third_party/python-peachpy' 2024-08-20T22:48:35.6783609Z Entering 'third_party/sleef' 2024-08-20T22:48:35.6835297Z Entering 'third_party/tensorpipe' 2024-08-20T22:48:35.6887550Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:48:35.6935480Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:48:35.6986624Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:48:35.7035266Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:48:35.7084159Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:48:35.7150322Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:48:35.7180118Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7186915Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:48:35.7219782Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:48:35.7506406Z Entering 'android/libs/fbjni' 2024-08-20T22:48:35.7539376Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7572402Z Entering 'third_party/FP16' 2024-08-20T22:48:35.7607269Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7640003Z Entering 'third_party/FXdiv' 2024-08-20T22:48:35.7674690Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7706538Z Entering 'third_party/NNPACK' 2024-08-20T22:48:35.7740296Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7773243Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:48:35.7807054Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7855746Z Entering 'third_party/XNNPACK' 2024-08-20T22:48:35.7874189Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7926937Z Entering 'third_party/benchmark' 2024-08-20T22:48:35.7960866Z http.https://github.com/.extraheader 2024-08-20T22:48:35.7993833Z Entering 'third_party/cpp-httplib' 2024-08-20T22:48:35.8028742Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8060695Z Entering 'third_party/cpuinfo' 2024-08-20T22:48:35.8094707Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8130378Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:48:35.8164196Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8197698Z Entering 'third_party/cutlass' 2024-08-20T22:48:35.8232387Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8273250Z Entering 'third_party/eigen' 2024-08-20T22:48:35.8308327Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8342628Z Entering 'third_party/fbgemm' 2024-08-20T22:48:35.8376849Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8411295Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:48:35.8445867Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8478950Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:48:35.8513329Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8545966Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:48:35.8579039Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8618025Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:48:35.8651226Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8684982Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:48:35.8719061Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8750624Z Entering 'third_party/flatbuffers' 2024-08-20T22:48:35.8786692Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8826280Z Entering 'third_party/fmt' 2024-08-20T22:48:35.8859142Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8893670Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:48:35.8926394Z http.https://github.com/.extraheader 2024-08-20T22:48:35.8957727Z Entering 'third_party/gloo' 2024-08-20T22:48:35.8993988Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9026088Z Entering 'third_party/googletest' 2024-08-20T22:48:35.9059173Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9093695Z Entering 'third_party/ideep' 2024-08-20T22:48:35.9128269Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9156304Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:48:35.9190889Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9231409Z Entering 'third_party/ittapi' 2024-08-20T22:48:35.9265927Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9297232Z Entering 'third_party/kineto' 2024-08-20T22:48:35.9331540Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9363942Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:48:35.9397304Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9429814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:48:35.9461826Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9495226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:48:35.9529709Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9561951Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:48:35.9595984Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9627918Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:48:35.9660795Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9694247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:48:35.9727235Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9762510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:48:35.9795319Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9832886Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:48:35.9866396Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9903018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:48:35.9935185Z http.https://github.com/.extraheader 2024-08-20T22:48:35.9968812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:48:36.0002240Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0035327Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:48:36.0069707Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0100337Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:48:36.0133885Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0166858Z Entering 'third_party/mimalloc' 2024-08-20T22:48:36.0201663Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0233363Z Entering 'third_party/nccl/nccl' 2024-08-20T22:48:36.0267609Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0298212Z Entering 'third_party/nlohmann' 2024-08-20T22:48:36.0332743Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0364365Z Entering 'third_party/onnx' 2024-08-20T22:48:36.0398547Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0444921Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:48:36.0478699Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0510444Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:48:36.0545778Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0583553Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:48:36.0617752Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0652727Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:48:36.0687005Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0718694Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:48:36.0750664Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0782602Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:48:36.0814540Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0850164Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:48:36.0884624Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0917147Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:48:36.0948775Z http.https://github.com/.extraheader 2024-08-20T22:48:36.0983434Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:48:36.1015272Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1050771Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:48:36.1085516Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1117378Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:48:36.1149765Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1189390Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:48:36.1220459Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1257744Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:48:36.1292403Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1349134Z Entering 'third_party/pocketfft' 2024-08-20T22:48:36.1381643Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1413884Z Entering 'third_party/protobuf' 2024-08-20T22:48:36.1447947Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1482812Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:48:36.1515578Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1546822Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:48:36.1579732Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1614279Z Entering 'third_party/psimd' 2024-08-20T22:48:36.1647556Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1680840Z Entering 'third_party/pthreadpool' 2024-08-20T22:48:36.1714577Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1753762Z Entering 'third_party/pybind11' 2024-08-20T22:48:36.1788515Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1819160Z Entering 'third_party/python-peachpy' 2024-08-20T22:48:36.1853610Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1887228Z Entering 'third_party/sleef' 2024-08-20T22:48:36.1920625Z http.https://github.com/.extraheader 2024-08-20T22:48:36.1950809Z Entering 'third_party/tensorpipe' 2024-08-20T22:48:36.1985891Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2020053Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:48:36.2052474Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2085225Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:48:36.2118140Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2147681Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:48:36.2180272Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2215431Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:48:36.2247316Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2277813Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:48:36.2311055Z http.https://github.com/.extraheader 2024-08-20T22:48:36.2434505Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:48:36.2451148Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:48:36.2456710Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:48:36.2457221Z ##[endgroup] 2024-08-20T22:48:42.5537676Z Cleaning up orphan processes