2024-06-01T03:28:12.5788767Z Current runner version: '2.317.0' 2024-06-01T03:28:12.5794861Z Runner name: 'i-0b128c63f91218fe4' 2024-06-01T03:28:12.5795645Z Runner group name: 'Default' 2024-06-01T03:28:12.5796525Z Machine name: 'ip-10-0-52-128' 2024-06-01T03:28:12.5801178Z ##[group]GITHUB_TOKEN Permissions 2024-06-01T03:28:12.5803169Z Actions: read 2024-06-01T03:28:12.5803684Z Attestations: read 2024-06-01T03:28:12.5804151Z Checks: read 2024-06-01T03:28:12.5804629Z Contents: read 2024-06-01T03:28:12.5805097Z Deployments: read 2024-06-01T03:28:12.5805591Z Discussions: read 2024-06-01T03:28:12.5806071Z Issues: read 2024-06-01T03:28:12.5806538Z Metadata: read 2024-06-01T03:28:12.5807008Z Packages: read 2024-06-01T03:28:12.5807468Z Pages: read 2024-06-01T03:28:12.5807923Z PullRequests: read 2024-06-01T03:28:12.5808449Z RepositoryProjects: read 2024-06-01T03:28:12.5809027Z SecurityEvents: read 2024-06-01T03:28:12.5809539Z Statuses: read 2024-06-01T03:28:12.5810001Z ##[endgroup] 2024-06-01T03:28:12.5813046Z Secret source: Actions 2024-06-01T03:28:12.5813800Z Prepare workflow directory 2024-06-01T03:28:12.6724918Z Prepare all required actions 2024-06-01T03:28:12.6889005Z Getting action download info 2024-06-01T03:28:12.8341698Z Download action repository 'pytorch/test-infra@main' (SHA:2a9c4e97b8b1ebef43b53b7f988220e3f225ca55) 2024-06-01T03:28:13.1488496Z Download action repository 'pytorch/pytorch@main' (SHA:25447ba241b788eb942af6f93c1dac71deadee65) 2024-06-01T03:28:16.0619900Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-01T03:28:16.2328198Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-01T03:28:16.4671452Z Getting action download info 2024-06-01T03:28:16.5558871Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-01T03:28:16.7075128Z Getting action download info 2024-06-01T03:28:16.7897442Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-01T03:28:16.9100050Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/127669 (de352ff31081bc3b80baf4f72168a00bdf6cccae) 2024-06-01T03:28:16.9102331Z ##[group] Inputs 2024-06-01T03:28:16.9102835Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:28:16.9111623Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-01T03:28:16.9121004Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:28:16.9122296Z sync-tag: 2024-06-01T03:28:16.9123223Z timeout-minutes: 240 2024-06-01T03:28:16.9123583Z use-gha: 2024-06-01T03:28:16.9123873Z dashboard-tag: 2024-06-01T03:28:16.9124215Z s3-bucket: gha-artifacts 2024-06-01T03:28:16.9124598Z aws-role-to-assume: 2024-06-01T03:28:16.9124951Z ##[endgroup] 2024-06-01T03:28:16.9125911Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:28:16.9699368Z A job started hook has been configured by the self-hosted runner administrator 2024-06-01T03:28:16.9865232Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-01T03:28:16.9876248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:28:16.9876805Z ##[endgroup] 2024-06-01T03:28:17.1273999Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-01T03:28:17.1274658Z with: 2024-06-01T03:28:17.1275367Z github-secret: *** 2024-06-01T03:28:17.1276448Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-01T03:28:17.1277555Z activate-with-label: false 2024-06-01T03:28:17.1277999Z label: with-ssh 2024-06-01T03:28:17.1278422Z remove-existing-keys: true 2024-06-01T03:28:17.1278914Z env: 2024-06-01T03:28:17.1279273Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:28:17.1279701Z ##[endgroup] 2024-06-01T03:28:17.2141518Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-01T03:28:17.2142689Z ciflow reference detected, attempting to extract PR number 2024-06-01T03:28:17.6153846Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-01T03:28:17.6826840Z No SSH keys found for user pytorch-bot[bot] 2024-06-01T03:28:17.6827591Z Grabbing public ssh keys from https://github.com/Fuzzkatt.keys 2024-06-01T03:28:17.7496775Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-06-01T03:28:17.7510335Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-06-01T03:28:17.7533297Z Login using: ssh ec2-user@ec2-54-85-163-109.compute-1.amazonaws.com 2024-06-01T03:28:17.7534134Z All testing is done inside the container, to start an interactive session run: 2024-06-01T03:28:17.7535089Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-01T03:28:17.7697298Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-01T03:28:17.7697946Z with: 2024-06-01T03:28:17.7698231Z submodules: recursive 2024-06-01T03:28:17.7698569Z fetch-depth: 0 2024-06-01T03:28:17.7698865Z env: 2024-06-01T03:28:17.7699147Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:28:17.7699488Z ##[endgroup] 2024-06-01T03:28:17.7891691Z ##[group]Run retry () { 2024-06-01T03:28:17.7892097Z retry () { 2024-06-01T03:28:17.7892627Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-01T03:28:17.7893200Z } 2024-06-01T03:28:17.7893519Z echo "${GITHUB_WORKSPACE}" 2024-06-01T03:28:17.7893958Z if [ -z "${NO_SUDO}" ]; then 2024-06-01T03:28:17.7894458Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-01T03:28:17.7894935Z else 2024-06-01T03:28:17.7895288Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-01T03:28:17.7895731Z fi 2024-06-01T03:28:17.7896088Z mkdir "${GITHUB_WORKSPACE}" 2024-06-01T03:28:17.7903594Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:28:17.7904230Z env: 2024-06-01T03:28:17.7904517Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:28:17.7904873Z NO_SUDO: 2024-06-01T03:28:17.7905147Z ##[endgroup] 2024-06-01T03:28:17.7926703Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T03:28:17.8108637Z ##[group]Run malfet/checkout@silent-checkout 2024-06-01T03:28:17.8109231Z with: 2024-06-01T03:28:17.8109566Z ref: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:28:17.8110361Z fetch-depth: 0 2024-06-01T03:28:17.8110701Z submodules: recursive 2024-06-01T03:28:17.8111051Z quiet-checkout: true 2024-06-01T03:28:17.8111412Z repository: pytorch/pytorch 2024-06-01T03:28:17.8111947Z token: *** 2024-06-01T03:28:17.8112256Z ssh-strict: true 2024-06-01T03:28:17.8112589Z persist-credentials: true 2024-06-01T03:28:17.8112959Z clean: true 2024-06-01T03:28:17.8113309Z sparse-checkout-cone-mode: true 2024-06-01T03:28:17.8113710Z lfs: false 2024-06-01T03:28:17.8114025Z set-safe-directory: true 2024-06-01T03:28:17.8114373Z env: 2024-06-01T03:28:17.8114657Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:28:17.8115006Z ##[endgroup] 2024-06-01T03:28:17.9027272Z Syncing repository: pytorch/pytorch 2024-06-01T03:28:17.9028899Z ##[group]Getting Git version info 2024-06-01T03:28:17.9029821Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-01T03:28:17.9031003Z [command]/usr/bin/git version 2024-06-01T03:28:17.9031385Z git version 2.40.1 2024-06-01T03:28:17.9032677Z ##[endgroup] 2024-06-01T03:28:17.9044887Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4344d9e2-e3cb-48a0-95b4-58b93b693cc1' before making global git config changes 2024-06-01T03:28:17.9046258Z Adding repository directory to the temporary git global config as a safe directory 2024-06-01T03:28:17.9047521Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T03:28:17.9066450Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-01T03:28:17.9070409Z ##[group]Initializing the repository 2024-06-01T03:28:17.9073644Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T03:28:17.9092994Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-01T03:28:17.9094025Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-01T03:28:17.9094870Z hint: of your new repositories, which will suppress this warning, call: 2024-06-01T03:28:17.9095471Z hint: 2024-06-01T03:28:17.9095931Z hint: git config --global init.defaultBranch 2024-06-01T03:28:17.9096425Z hint: 2024-06-01T03:28:17.9096948Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-01T03:28:17.9097831Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-01T03:28:17.9098611Z hint: 2024-06-01T03:28:17.9098944Z hint: git branch -m 2024-06-01T03:28:17.9099713Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-01T03:28:17.9102385Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-01T03:28:17.9126097Z ##[endgroup] 2024-06-01T03:28:17.9126956Z ##[group]Disabling automatic garbage collection 2024-06-01T03:28:17.9130472Z [command]/usr/bin/git config --local gc.auto 0 2024-06-01T03:28:17.9149886Z ##[endgroup] 2024-06-01T03:28:17.9150988Z ##[group]Setting up auth 2024-06-01T03:28:17.9157428Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-01T03:28:17.9180698Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-01T03:28:17.9383574Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-01T03:28:17.9404712Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-01T03:28:17.9596573Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-01T03:28:17.9632953Z ##[endgroup] 2024-06-01T03:28:17.9633533Z ##[group]Fetching the repository 2024-06-01T03:28:17.9639042Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-01T03:28:21.5799418Z remote: Enumerating objects: 1124709 2024-06-01T03:28:21.5800191Z remote: Enumerating objects: 1128875, done. 2024-06-01T03:28:21.5806342Z remote: Counting objects: 0% (1/4166) 2024-06-01T03:28:21.5810520Z remote: Counting objects: 1% (42/4166) 2024-06-01T03:28:21.5812530Z remote: Counting objects: 2% (84/4166) 2024-06-01T03:28:21.5813277Z remote: Counting objects: 3% (125/4166) 2024-06-01T03:28:21.5815506Z remote: Counting objects: 4% (167/4166) 2024-06-01T03:28:21.5816278Z remote: Counting objects: 5% (209/4166) 2024-06-01T03:28:21.5818083Z remote: Counting objects: 6% (250/4166) 2024-06-01T03:28:21.5820839Z remote: Counting objects: 7% (292/4166) 2024-06-01T03:28:21.5821571Z remote: Counting objects: 8% (334/4166) 2024-06-01T03:28:21.5823818Z remote: Counting objects: 9% (375/4166) 2024-06-01T03:28:21.5825194Z remote: Counting objects: 10% (417/4166) 2024-06-01T03:28:21.5826740Z remote: Counting objects: 11% (459/4166) 2024-06-01T03:28:21.5827423Z remote: Counting objects: 12% (500/4166) 2024-06-01T03:28:21.5830257Z remote: Counting objects: 13% (542/4166) 2024-06-01T03:28:21.5830793Z remote: Counting objects: 14% (584/4166) 2024-06-01T03:28:21.5832635Z remote: Counting objects: 15% (625/4166) 2024-06-01T03:28:21.5834334Z remote: Counting objects: 16% (667/4166) 2024-06-01T03:28:21.5834856Z remote: Counting objects: 17% (709/4166) 2024-06-01T03:28:21.5836541Z remote: Counting objects: 18% (750/4166) 2024-06-01T03:28:21.5837214Z remote: Counting objects: 19% (792/4166) 2024-06-01T03:28:21.5838707Z remote: Counting objects: 20% (834/4166) 2024-06-01T03:28:21.5841014Z remote: Counting objects: 21% (875/4166) 2024-06-01T03:28:21.5841638Z remote: Counting objects: 22% (917/4166) 2024-06-01T03:28:21.5842262Z remote: Counting objects: 23% (959/4166) 2024-06-01T03:28:21.5843751Z remote: Counting objects: 24% (1000/4166) 2024-06-01T03:28:21.5845218Z remote: Counting objects: 25% (1042/4166) 2024-06-01T03:28:21.5845771Z remote: Counting objects: 26% (1084/4166) 2024-06-01T03:28:21.5846683Z remote: Counting objects: 27% (1125/4166) 2024-06-01T03:28:21.5849211Z remote: Counting objects: 28% (1167/4166) 2024-06-01T03:28:21.5850743Z remote: Counting objects: 29% (1209/4166) 2024-06-01T03:28:21.5851723Z remote: Counting objects: 30% (1250/4166) 2024-06-01T03:28:21.5852418Z remote: Counting objects: 31% (1292/4166) 2024-06-01T03:28:21.5857027Z remote: Counting objects: 32% (1334/4166) 2024-06-01T03:28:21.5857765Z remote: Counting objects: 33% (1375/4166) 2024-06-01T03:28:21.5858346Z remote: Counting objects: 34% (1417/4166) 2024-06-01T03:28:21.5858927Z remote: Counting objects: 35% (1459/4166) 2024-06-01T03:28:21.5859653Z remote: Counting objects: 36% (1500/4166) 2024-06-01T03:28:21.5860316Z remote: Counting objects: 37% (1542/4166) 2024-06-01T03:28:21.5860844Z remote: Counting objects: 38% (1584/4166) 2024-06-01T03:28:21.5861367Z remote: Counting objects: 39% (1625/4166) 2024-06-01T03:28:21.5861899Z remote: Counting objects: 40% (1667/4166) 2024-06-01T03:28:21.5862523Z remote: Counting objects: 41% (1709/4166) 2024-06-01T03:28:21.5863206Z remote: Counting objects: 42% (1750/4166) 2024-06-01T03:28:21.5863813Z remote: Counting objects: 43% (1792/4166) 2024-06-01T03:28:21.5864351Z remote: Counting objects: 44% (1834/4166) 2024-06-01T03:28:21.5864874Z remote: Counting objects: 45% (1875/4166) 2024-06-01T03:28:21.5865480Z remote: Counting objects: 46% (1917/4166) 2024-06-01T03:28:21.5866015Z remote: Counting objects: 47% (1959/4166) 2024-06-01T03:28:21.5866552Z remote: Counting objects: 48% (2000/4166) 2024-06-01T03:28:21.5867086Z remote: Counting objects: 49% (2042/4166) 2024-06-01T03:28:21.5867639Z remote: Counting objects: 50% (2083/4166) 2024-06-01T03:28:21.5868243Z remote: Counting objects: 51% (2125/4166) 2024-06-01T03:28:21.5868762Z remote: Counting objects: 52% (2167/4166) 2024-06-01T03:28:21.5869294Z remote: Counting objects: 53% (2208/4166) 2024-06-01T03:28:21.5870125Z remote: Counting objects: 54% (2250/4166) 2024-06-01T03:28:21.5870788Z remote: Counting objects: 55% (2292/4166) 2024-06-01T03:28:21.5871314Z remote: Counting objects: 56% (2333/4166) 2024-06-01T03:28:21.5871841Z remote: Counting objects: 57% (2375/4166) 2024-06-01T03:28:21.5872447Z remote: Counting objects: 58% (2417/4166) 2024-06-01T03:28:21.5873154Z remote: Counting objects: 59% (2458/4166) 2024-06-01T03:28:21.5873696Z remote: Counting objects: 60% (2500/4166) 2024-06-01T03:28:21.5874227Z remote: Counting objects: 61% (2542/4166) 2024-06-01T03:28:21.5874929Z remote: Counting objects: 62% (2583/4166) 2024-06-01T03:28:21.5875557Z remote: Counting objects: 63% (2625/4166) 2024-06-01T03:28:21.5878458Z remote: Counting objects: 64% (2667/4166) 2024-06-01T03:28:21.5879168Z remote: Counting objects: 65% (2708/4166) 2024-06-01T03:28:21.5879871Z remote: Counting objects: 66% (2750/4166) 2024-06-01T03:28:21.5880407Z remote: Counting objects: 67% (2792/4166) 2024-06-01T03:28:21.5880963Z remote: Counting objects: 68% (2833/4166) 2024-06-01T03:28:21.5881489Z remote: Counting objects: 69% (2875/4166) 2024-06-01T03:28:21.5882177Z remote: Counting objects: 70% (2917/4166) 2024-06-01T03:28:21.5882706Z remote: Counting objects: 71% (2958/4166) 2024-06-01T03:28:21.5883236Z remote: Counting objects: 72% (3000/4166) 2024-06-01T03:28:21.5883756Z remote: Counting objects: 73% (3042/4166) 2024-06-01T03:28:21.5884281Z remote: Counting objects: 74% (3083/4166) 2024-06-01T03:28:21.5884804Z remote: Counting objects: 75% (3125/4166) 2024-06-01T03:28:21.5885330Z remote: Counting objects: 76% (3167/4166) 2024-06-01T03:28:21.5885850Z remote: Counting objects: 77% (3208/4166) 2024-06-01T03:28:21.5886382Z remote: Counting objects: 78% (3250/4166) 2024-06-01T03:28:21.5887024Z remote: Counting objects: 79% (3292/4166) 2024-06-01T03:28:21.5887607Z remote: Counting objects: 80% (3333/4166) 2024-06-01T03:28:21.5888133Z remote: Counting objects: 81% (3375/4166) 2024-06-01T03:28:21.5888666Z remote: Counting objects: 82% (3417/4166) 2024-06-01T03:28:21.5889192Z remote: Counting objects: 83% (3458/4166) 2024-06-01T03:28:21.5889716Z remote: Counting objects: 84% (3500/4166) 2024-06-01T03:28:21.5890234Z remote: Counting objects: 85% (3542/4166) 2024-06-01T03:28:21.5890758Z remote: Counting objects: 86% (3583/4166) 2024-06-01T03:28:21.5891285Z remote: Counting objects: 87% (3625/4166) 2024-06-01T03:28:21.5891807Z remote: Counting objects: 88% (3667/4166) 2024-06-01T03:28:21.5892323Z remote: Counting objects: 89% (3708/4166) 2024-06-01T03:28:21.5892848Z remote: Counting objects: 90% (3750/4166) 2024-06-01T03:28:21.5893378Z remote: Counting objects: 91% (3792/4166) 2024-06-01T03:28:21.5893904Z remote: Counting objects: 92% (3833/4166) 2024-06-01T03:28:21.5894490Z remote: Counting objects: 93% (3875/4166) 2024-06-01T03:28:21.5895016Z remote: Counting objects: 94% (3917/4166) 2024-06-01T03:28:21.5895539Z remote: Counting objects: 95% (3958/4166) 2024-06-01T03:28:21.5896065Z remote: Counting objects: 96% (4000/4166) 2024-06-01T03:28:21.5896584Z remote: Counting objects: 97% (4042/4166) 2024-06-01T03:28:21.5897113Z remote: Counting objects: 98% (4083/4166) 2024-06-01T03:28:21.5897782Z remote: Counting objects: 99% (4125/4166) 2024-06-01T03:28:21.5898355Z remote: Counting objects: 100% (4166/4166) 2024-06-01T03:28:21.5898951Z remote: Counting objects: 100% (4166/4166), done. 2024-06-01T03:28:21.6286645Z remote: Compressing objects: 0% (1/2079) 2024-06-01T03:28:21.7089787Z remote: Compressing objects: 1% (21/2079) 2024-06-01T03:28:21.7367200Z remote: Compressing objects: 2% (42/2079) 2024-06-01T03:28:21.8249874Z remote: Compressing objects: 3% (63/2079) 2024-06-01T03:28:21.8880040Z remote: Compressing objects: 4% (84/2079) 2024-06-01T03:28:22.2334294Z remote: Compressing objects: 5% (104/2079) 2024-06-01T03:28:22.4565212Z remote: Compressing objects: 6% (125/2079) 2024-06-01T03:28:22.5838529Z remote: Compressing objects: 7% (146/2079) 2024-06-01T03:28:22.5897701Z remote: Compressing objects: 8% (167/2079) 2024-06-01T03:28:22.6773290Z remote: Compressing objects: 8% (169/2079) 2024-06-01T03:28:22.7488761Z remote: Compressing objects: 9% (188/2079) 2024-06-01T03:28:22.8130996Z remote: Compressing objects: 10% (208/2079) 2024-06-01T03:28:22.8650387Z remote: Compressing objects: 11% (229/2079) 2024-06-01T03:28:22.8973726Z remote: Compressing objects: 12% (250/2079) 2024-06-01T03:28:22.9319613Z remote: Compressing objects: 13% (271/2079) 2024-06-01T03:28:22.9513940Z remote: Compressing objects: 14% (292/2079) 2024-06-01T03:28:22.9643195Z remote: Compressing objects: 15% (312/2079) 2024-06-01T03:28:22.9761256Z remote: Compressing objects: 16% (333/2079) 2024-06-01T03:28:22.9830058Z remote: Compressing objects: 17% (354/2079) 2024-06-01T03:28:22.9847951Z remote: Compressing objects: 18% (375/2079) 2024-06-01T03:28:22.9882635Z remote: Compressing objects: 19% (396/2079) 2024-06-01T03:28:22.9924079Z remote: Compressing objects: 20% (416/2079) 2024-06-01T03:28:22.9981563Z remote: Compressing objects: 21% (437/2079) 2024-06-01T03:28:23.0015481Z remote: Compressing objects: 22% (458/2079) 2024-06-01T03:28:23.0041050Z remote: Compressing objects: 23% (479/2079) 2024-06-01T03:28:23.0058270Z remote: Compressing objects: 24% (499/2079) 2024-06-01T03:28:23.0072288Z remote: Compressing objects: 25% (520/2079) 2024-06-01T03:28:23.0083378Z remote: Compressing objects: 26% (541/2079) 2024-06-01T03:28:23.0090008Z remote: Compressing objects: 27% (562/2079) 2024-06-01T03:28:23.0107493Z remote: Compressing objects: 28% (583/2079) 2024-06-01T03:28:23.0125518Z remote: Compressing objects: 29% (603/2079) 2024-06-01T03:28:23.0159545Z remote: Compressing objects: 30% (624/2079) 2024-06-01T03:28:23.0193836Z remote: Compressing objects: 31% (645/2079) 2024-06-01T03:28:23.0213921Z remote: Compressing objects: 32% (666/2079) 2024-06-01T03:28:23.0247985Z remote: Compressing objects: 33% (687/2079) 2024-06-01T03:28:23.0288394Z remote: Compressing objects: 34% (707/2079) 2024-06-01T03:28:23.0315418Z remote: Compressing objects: 35% (728/2079) 2024-06-01T03:28:23.0342882Z remote: Compressing objects: 36% (749/2079) 2024-06-01T03:28:23.0359300Z remote: Compressing objects: 37% (770/2079) 2024-06-01T03:28:23.0389580Z remote: Compressing objects: 38% (791/2079) 2024-06-01T03:28:23.0421892Z remote: Compressing objects: 39% (811/2079) 2024-06-01T03:28:23.0441162Z remote: Compressing objects: 40% (832/2079) 2024-06-01T03:28:23.0468373Z remote: Compressing objects: 41% (853/2079) 2024-06-01T03:28:23.0502457Z remote: Compressing objects: 42% (874/2079) 2024-06-01T03:28:23.0524859Z remote: Compressing objects: 43% (894/2079) 2024-06-01T03:28:23.0540442Z remote: Compressing objects: 44% (915/2079) 2024-06-01T03:28:23.0563879Z remote: Compressing objects: 45% (936/2079) 2024-06-01T03:28:23.0590219Z remote: Compressing objects: 46% (957/2079) 2024-06-01T03:28:23.0601867Z remote: Compressing objects: 47% (978/2079) 2024-06-01T03:28:23.0624072Z remote: Compressing objects: 48% (998/2079) 2024-06-01T03:28:23.0633383Z remote: Compressing objects: 49% (1019/2079) 2024-06-01T03:28:23.0651397Z remote: Compressing objects: 50% (1040/2079) 2024-06-01T03:28:23.0662043Z remote: Compressing objects: 51% (1061/2079) 2024-06-01T03:28:23.0680180Z remote: Compressing objects: 52% (1082/2079) 2024-06-01T03:28:23.0702132Z remote: Compressing objects: 53% (1102/2079) 2024-06-01T03:28:23.0720537Z remote: Compressing objects: 54% (1123/2079) 2024-06-01T03:28:23.0723672Z remote: Compressing objects: 55% (1144/2079) 2024-06-01T03:28:23.0738054Z remote: Compressing objects: 56% (1165/2079) 2024-06-01T03:28:23.0750178Z remote: Compressing objects: 57% (1186/2079) 2024-06-01T03:28:23.0758861Z remote: Compressing objects: 58% (1206/2079) 2024-06-01T03:28:23.0766918Z remote: Compressing objects: 59% (1227/2079) 2024-06-01T03:28:23.0775709Z remote: Compressing objects: 60% (1248/2079) 2024-06-01T03:28:23.0780821Z remote: Compressing objects: 61% (1269/2079) 2024-06-01T03:28:23.0789135Z remote: Compressing objects: 62% (1289/2079) 2024-06-01T03:28:23.0794143Z remote: Compressing objects: 63% (1310/2079) 2024-06-01T03:28:23.0798872Z remote: Compressing objects: 64% (1331/2079) 2024-06-01T03:28:23.0802458Z remote: Compressing objects: 65% (1352/2079) 2024-06-01T03:28:23.0804434Z remote: Compressing objects: 66% (1373/2079) 2024-06-01T03:28:23.0807120Z remote: Compressing objects: 67% (1393/2079) 2024-06-01T03:28:23.0807722Z remote: Compressing objects: 68% (1414/2079) 2024-06-01T03:28:23.0808623Z remote: Compressing objects: 69% (1435/2079) 2024-06-01T03:28:23.0810538Z remote: Compressing objects: 70% (1456/2079) 2024-06-01T03:28:23.0830227Z remote: Compressing objects: 71% (1477/2079) 2024-06-01T03:28:23.0856324Z remote: Compressing objects: 72% (1497/2079) 2024-06-01T03:28:23.0868383Z remote: Compressing objects: 73% (1518/2079) 2024-06-01T03:28:23.0876320Z remote: Compressing objects: 74% (1539/2079) 2024-06-01T03:28:23.0880290Z remote: Compressing objects: 75% (1560/2079) 2024-06-01T03:28:23.0888230Z remote: Compressing objects: 76% (1581/2079) 2024-06-01T03:28:23.0896369Z remote: Compressing objects: 77% (1601/2079) 2024-06-01T03:28:23.0903964Z remote: Compressing objects: 78% (1622/2079) 2024-06-01T03:28:23.0915285Z remote: Compressing objects: 79% (1643/2079) 2024-06-01T03:28:23.0932517Z remote: Compressing objects: 80% (1664/2079) 2024-06-01T03:28:23.0943244Z remote: Compressing objects: 81% (1684/2079) 2024-06-01T03:28:23.0943875Z remote: Compressing objects: 82% (1705/2079) 2024-06-01T03:28:23.0950906Z remote: Compressing objects: 83% (1726/2079) 2024-06-01T03:28:23.0951538Z remote: Compressing objects: 84% (1747/2079) 2024-06-01T03:28:23.0959604Z remote: Compressing objects: 85% (1768/2079) 2024-06-01T03:28:23.0960259Z remote: Compressing objects: 86% (1788/2079) 2024-06-01T03:28:23.0964249Z remote: Compressing objects: 87% (1809/2079) 2024-06-01T03:28:23.0965114Z remote: Compressing objects: 88% (1830/2079) 2024-06-01T03:28:23.0965777Z remote: Compressing objects: 89% (1851/2079) 2024-06-01T03:28:23.0966610Z remote: Compressing objects: 90% (1872/2079) 2024-06-01T03:28:23.0973589Z remote: Compressing objects: 91% (1892/2079) 2024-06-01T03:28:23.0976172Z remote: Compressing objects: 92% (1913/2079) 2024-06-01T03:28:23.0976984Z remote: Compressing objects: 93% (1934/2079) 2024-06-01T03:28:23.0980277Z remote: Compressing objects: 94% (1955/2079) 2024-06-01T03:28:23.0983261Z remote: Compressing objects: 95% (1976/2079) 2024-06-01T03:28:23.0989572Z remote: Compressing objects: 96% (1996/2079) 2024-06-01T03:28:23.0992351Z remote: Compressing objects: 97% (2017/2079) 2024-06-01T03:28:23.0994370Z remote: Compressing objects: 98% (2038/2079) 2024-06-01T03:28:23.0995085Z remote: Compressing objects: 99% (2059/2079) 2024-06-01T03:28:23.0995704Z remote: Compressing objects: 100% (2079/2079) 2024-06-01T03:28:23.0996299Z remote: Compressing objects: 100% (2079/2079), done. 2024-06-01T03:28:45.0108607Z remote: Total 1128875 (delta 3051), reused 2925 (delta 2081), pack-reused 1124709 2024-06-01T03:29:11.4980783Z [command]/usr/bin/git rev-parse --verify --quiet de352ff31081bc3b80baf4f72168a00bdf6cccae^{object} 2024-06-01T03:29:11.4998074Z de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:29:11.5006367Z ##[endgroup] 2024-06-01T03:29:11.5007329Z ##[group]Determining the checkout info 2024-06-01T03:29:11.5008467Z ##[endgroup] 2024-06-01T03:29:11.5009300Z ##[group]Checking out the ref 2024-06-01T03:29:11.5010661Z [command]/usr/bin/git checkout --quiet --force de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:29:12.7057776Z ##[endgroup] 2024-06-01T03:29:12.7058483Z ##[group]Setting up auth for fetching submodules 2024-06-01T03:29:12.7063908Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-01T03:29:12.7110776Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-01T03:29:12.7129070Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-01T03:29:12.7146790Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-01T03:29:12.7162706Z ##[endgroup] 2024-06-01T03:29:12.7163248Z ##[group]Fetching submodules 2024-06-01T03:29:12.7166210Z [command]/usr/bin/git submodule sync --recursive 2024-06-01T03:29:12.7365061Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-01T03:29:12.7565334Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-01T03:29:12.7566851Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-01T03:29:12.7568691Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-01T03:29:12.7570960Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-01T03:29:12.7573762Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-01T03:29:12.7575641Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-01T03:29:12.7578308Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-01T03:29:12.7581015Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-01T03:29:12.7583576Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-01T03:29:12.7586385Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-01T03:29:12.7589047Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-01T03:29:12.7592233Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-01T03:29:12.7595074Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-01T03:29:12.7598120Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-01T03:29:12.7601029Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-01T03:29:12.7604296Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-01T03:29:12.7607650Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:29:12.7610665Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-01T03:29:12.7613995Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-01T03:29:12.7617184Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-01T03:29:12.7620788Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-06-01T03:29:12.7624179Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-01T03:29:12.7627860Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-01T03:29:12.7631789Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-01T03:29:12.7635503Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-01T03:29:12.7639196Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-01T03:29:12.7643062Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-01T03:29:12.7647429Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-01T03:29:12.7651046Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-01T03:29:12.7655160Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-01T03:29:12.7659318Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-01T03:29:12.7663521Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-01T03:29:12.7667529Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-01T03:29:12.7672215Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-01T03:29:12.7676334Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-01T03:29:12.7680887Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-01T03:29:12.7700838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-01T03:29:13.0146868Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-01T03:29:13.1688487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-01T03:29:13.3175040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-01T03:29:13.5328117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-01T03:29:15.6351734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-01T03:29:25.1601455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-01T03:29:25.5185328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-01T03:29:26.0077821Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-01T03:29:26.5428983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-01T03:29:27.7677914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-01T03:29:29.5349827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-01T03:29:35.4059157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-01T03:29:36.4914940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-01T03:29:38.1100415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-01T03:29:39.2357472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-01T03:29:39.3705197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-01T03:29:39.7480825Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-01T03:29:40.0269701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-01T03:29:40.9862294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-01T03:29:41.3872803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-06-01T03:29:41.5994634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-01T03:29:41.8373631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-01T03:29:43.2913452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-01T03:29:43.9751967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-01T03:29:44.5761592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-01T03:29:50.0916854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-01T03:29:52.4309103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-01T03:29:56.7933805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-01T03:29:56.9601402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-01T03:30:05.0917307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-01T03:30:05.2380424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-01T03:30:05.4050788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-01T03:30:06.2873634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-01T03:30:06.5642819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-01T03:30:07.1547710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-01T03:30:07.5700025Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-01T03:30:07.5782147Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-01T03:30:07.5842543Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-01T03:30:07.6022018Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-01T03:30:07.6323677Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-01T03:30:08.3894660Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-01T03:30:08.4057661Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-01T03:30:08.4399182Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-01T03:30:08.5213333Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-06-01T03:30:08.5459971Z Submodule path 'third_party/cudnn_frontend': checked out 'b740542818f36857acf7f9853f749bbad4118c65' 2024-06-01T03:30:08.9677270Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-01T03:30:09.1792220Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-01T03:30:09.2381911Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-01T03:30:09.2391930Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:30:09.2393901Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:30:09.2395888Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:30:09.2397492Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:30:09.2399460Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:30:09.2418449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-01T03:30:10.2317804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-01T03:30:10.7930182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-01T03:30:12.4761454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-01T03:30:13.4798774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-01T03:30:13.7979500Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-01T03:30:13.8812352Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-01T03:30:14.2189779Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-01T03:30:14.2725868Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-01T03:30:14.2809882Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-01T03:30:14.3712897Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-01T03:30:14.4017275Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-01T03:30:14.4081777Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-01T03:30:14.4384222Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-01T03:30:14.4580342Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-01T03:30:14.4943549Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-01T03:30:14.5039591Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-01T03:30:14.5050514Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-01T03:30:14.5066182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-01T03:30:26.2127627Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-01T03:30:26.2199627Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-06-01T03:30:26.2327349Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-01T03:30:26.3099797Z Submodule path 'third_party/kineto': checked out 'be1317644c68b4bfc4646024a6b221066e430031' 2024-06-01T03:30:26.3109907Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:30:26.3111836Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:30:26.3113877Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:30:26.3131023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-01T03:30:26.8271461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-01T03:30:27.9836686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-01T03:30:28.9713251Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-01T03:30:28.9723486Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:30:28.9725431Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:30:28.9727713Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:30:28.9729791Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:30:28.9731921Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:30:28.9734531Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:30:28.9736631Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:30:28.9739236Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:30:28.9756492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-01T03:30:29.9661721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-01T03:30:30.3163776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-01T03:30:31.5326049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-01T03:30:31.7788613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-01T03:30:32.1089181Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-01T03:30:33.0826003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-01T03:30:38.7640856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-01T03:30:39.1093951Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-01T03:30:39.1228693Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-01T03:30:39.1512064Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-01T03:30:39.1608593Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-01T03:30:39.1618931Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:30:39.1634888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-01T03:30:39.4241783Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-01T03:30:39.4372788Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-01T03:30:39.4705710Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-01T03:30:39.5504599Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-01T03:30:39.5621108Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-01T03:30:39.5904192Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-01T03:30:39.6393924Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-01T03:30:39.6680220Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-01T03:30:39.6874931Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-01T03:30:39.7717090Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-01T03:30:40.0412352Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-01T03:30:40.0435850Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-01T03:30:40.0437643Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-01T03:30:40.0452668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-01T03:30:40.4212957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-01T03:30:41.3395676Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-01T03:30:41.3680690Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-01T03:30:41.4201177Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-01T03:30:41.4213634Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:30:41.4215663Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:30:41.4217628Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:30:41.4219586Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:30:41.4221829Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:30:41.4224214Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:30:41.4226463Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:30:41.4228383Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:30:41.4245679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-01T03:30:41.7726672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-01T03:30:42.8220492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-01T03:30:43.1918466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-01T03:30:48.6478600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-01T03:30:48.9181939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-01T03:30:49.0896919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-01T03:30:49.4028563Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-01T03:30:55.2513300Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-01T03:30:55.2839370Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-01T03:30:55.2960708Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-01T03:30:55.3792314Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-01T03:30:55.3889053Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-01T03:30:55.3995375Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-01T03:30:55.4103422Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-01T03:30:55.4113751Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:30:55.4115755Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:30:55.4131853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-01T03:30:57.1362361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-01T03:30:58.2814615Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-01T03:30:58.3174346Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-01T03:30:58.6682677Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-01T03:30:58.6760069Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-01T03:30:58.8917295Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-01T03:30:58.8930789Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:30:58.8932477Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-01T03:30:58.8949758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-01T03:30:59.2521390Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-01T03:31:00.2031743Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-01T03:31:00.2653886Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-01T03:31:00.2713562Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-01T03:31:00.2798998Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-01T03:31:00.3071216Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-01T03:31:00.3291978Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-01T03:31:00.3619385Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-01T03:31:00.3819388Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-01T03:31:00.3829681Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:00.3831620Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:00.3833759Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:00.3835941Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:00.3853130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-01T03:31:01.3312053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-01T03:31:01.5741471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-01T03:31:02.6082733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-01T03:31:03.7785818Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-01T03:31:03.7896138Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-01T03:31:03.8454949Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-01T03:31:03.8681272Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-01T03:31:03.8690906Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:03.8708161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-01T03:31:04.0322662Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-01T03:31:04.0344592Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-01T03:31:04.0544705Z Entering 'android/libs/fbjni' 2024-06-01T03:31:04.0574061Z Entering 'third_party/FP16' 2024-06-01T03:31:04.0600009Z Entering 'third_party/FXdiv' 2024-06-01T03:31:04.0627706Z Entering 'third_party/NNPACK' 2024-06-01T03:31:04.0653525Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T03:31:04.0678276Z Entering 'third_party/XNNPACK' 2024-06-01T03:31:04.0714504Z Entering 'third_party/benchmark' 2024-06-01T03:31:04.0740562Z Entering 'third_party/cpp-httplib' 2024-06-01T03:31:04.0767199Z Entering 'third_party/cpuinfo' 2024-06-01T03:31:04.0794717Z Entering 'third_party/cudnn_frontend' 2024-06-01T03:31:04.0822998Z Entering 'third_party/cutlass' 2024-06-01T03:31:04.0853604Z Entering 'third_party/eigen' 2024-06-01T03:31:04.0883097Z Entering 'third_party/fbgemm' 2024-06-01T03:31:04.0911697Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:31:04.0936415Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:31:04.0961926Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:31:04.0993989Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:31:04.1021335Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:31:04.1048906Z Entering 'third_party/flatbuffers' 2024-06-01T03:31:04.1081329Z Entering 'third_party/fmt' 2024-06-01T03:31:04.1105629Z Entering 'third_party/foxi' 2024-06-01T03:31:04.1131350Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:31:04.1159327Z Entering 'third_party/gloo' 2024-06-01T03:31:04.1184032Z Entering 'third_party/googletest' 2024-06-01T03:31:04.1210680Z Entering 'third_party/ideep' 2024-06-01T03:31:04.1238133Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T03:31:04.1271506Z Entering 'third_party/ios-cmake' 2024-06-01T03:31:04.1298042Z Entering 'third_party/ittapi' 2024-06-01T03:31:04.1325776Z Entering 'third_party/kineto' 2024-06-01T03:31:04.1352831Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:31:04.1380264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:31:04.1408583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:31:04.1435616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:31:04.1463419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:31:04.1489518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:31:04.1517564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:31:04.1544206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:31:04.1572286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:31:04.1597390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:31:04.1624138Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:31:04.1649327Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:31:04.1677919Z Entering 'third_party/mimalloc' 2024-06-01T03:31:04.1706473Z Entering 'third_party/nccl/nccl' 2024-06-01T03:31:04.1735329Z Entering 'third_party/nlohmann' 2024-06-01T03:31:04.1762455Z Entering 'third_party/onnx' 2024-06-01T03:31:04.1801585Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T03:31:04.1830477Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T03:31:04.1859776Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T03:31:04.1888579Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:31:04.1915130Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:31:04.1940426Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:31:04.1968278Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:31:04.1994734Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:31:04.2020928Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:31:04.2047316Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:31:04.2072256Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:31:04.2100090Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:31:04.2129156Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:31:04.2170072Z Entering 'third_party/pocketfft' 2024-06-01T03:31:04.2198231Z Entering 'third_party/protobuf' 2024-06-01T03:31:04.2227458Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:31:04.2255399Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T03:31:04.2282063Z Entering 'third_party/psimd' 2024-06-01T03:31:04.2310389Z Entering 'third_party/pthreadpool' 2024-06-01T03:31:04.2338626Z Entering 'third_party/pybind11' 2024-06-01T03:31:04.2366737Z Entering 'third_party/python-peachpy' 2024-06-01T03:31:04.2394781Z Entering 'third_party/sleef' 2024-06-01T03:31:04.2423376Z Entering 'third_party/tensorpipe' 2024-06-01T03:31:04.2449183Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:04.2476047Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:04.2501623Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:04.2527427Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:04.2551600Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:04.2592878Z ##[endgroup] 2024-06-01T03:31:04.2594707Z ##[group]Persisting credentials for submodules 2024-06-01T03:31:04.2596828Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-01T03:31:04.2799292Z Entering 'android/libs/fbjni' 2024-06-01T03:31:04.2832403Z Entering 'third_party/FP16' 2024-06-01T03:31:04.2864007Z Entering 'third_party/FXdiv' 2024-06-01T03:31:04.2897297Z Entering 'third_party/NNPACK' 2024-06-01T03:31:04.2932093Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T03:31:04.2966255Z Entering 'third_party/XNNPACK' 2024-06-01T03:31:04.3010088Z Entering 'third_party/benchmark' 2024-06-01T03:31:04.3046440Z Entering 'third_party/cpp-httplib' 2024-06-01T03:31:04.3078003Z Entering 'third_party/cpuinfo' 2024-06-01T03:31:04.3112184Z Entering 'third_party/cudnn_frontend' 2024-06-01T03:31:04.3146408Z Entering 'third_party/cutlass' 2024-06-01T03:31:04.3187598Z Entering 'third_party/eigen' 2024-06-01T03:31:04.3225101Z Entering 'third_party/fbgemm' 2024-06-01T03:31:04.3260470Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:31:04.3295811Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:31:04.3329252Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:31:04.3367122Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:31:04.3398179Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:31:04.3431593Z Entering 'third_party/flatbuffers' 2024-06-01T03:31:04.3467341Z Entering 'third_party/fmt' 2024-06-01T03:31:04.3502263Z Entering 'third_party/foxi' 2024-06-01T03:31:04.3538012Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:31:04.3570949Z Entering 'third_party/gloo' 2024-06-01T03:31:04.3606202Z Entering 'third_party/googletest' 2024-06-01T03:31:04.3641743Z Entering 'third_party/ideep' 2024-06-01T03:31:04.3676421Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T03:31:04.3713596Z Entering 'third_party/ios-cmake' 2024-06-01T03:31:04.3746946Z Entering 'third_party/ittapi' 2024-06-01T03:31:04.3780960Z Entering 'third_party/kineto' 2024-06-01T03:31:04.3816757Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:31:04.3848360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:31:04.3883712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:31:04.3917687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:31:04.3952210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:31:04.3983986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:31:04.4020151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:31:04.4053760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:31:04.4086063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:31:04.4122116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:31:04.4158042Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:31:04.4191364Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:31:04.4226052Z Entering 'third_party/mimalloc' 2024-06-01T03:31:04.4262057Z Entering 'third_party/nccl/nccl' 2024-06-01T03:31:04.4298216Z Entering 'third_party/nlohmann' 2024-06-01T03:31:04.4331540Z Entering 'third_party/onnx' 2024-06-01T03:31:04.4379202Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T03:31:04.4413104Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T03:31:04.4449559Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T03:31:04.4485800Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:31:04.4515647Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:31:04.4548217Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:31:04.4583580Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:31:04.4618704Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:31:04.4654432Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:31:04.4686691Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:31:04.4719185Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:31:04.4754494Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:31:04.4789599Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:31:04.4837479Z Entering 'third_party/pocketfft' 2024-06-01T03:31:04.4870217Z Entering 'third_party/protobuf' 2024-06-01T03:31:04.4905393Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:31:04.4938912Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T03:31:04.4973410Z Entering 'third_party/psimd' 2024-06-01T03:31:04.5008915Z Entering 'third_party/pthreadpool' 2024-06-01T03:31:04.5043165Z Entering 'third_party/pybind11' 2024-06-01T03:31:04.5077912Z Entering 'third_party/python-peachpy' 2024-06-01T03:31:04.5113054Z Entering 'third_party/sleef' 2024-06-01T03:31:04.5145012Z Entering 'third_party/tensorpipe' 2024-06-01T03:31:04.5178679Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:04.5211789Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:04.5244947Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:04.5278486Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:04.5310306Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:04.5358041Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-01T03:31:04.5557030Z Entering 'android/libs/fbjni' 2024-06-01T03:31:04.5588980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-01T03:31:04.5601720Z Entering 'third_party/FP16' 2024-06-01T03:31:04.5630884Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-01T03:31:04.5643130Z Entering 'third_party/FXdiv' 2024-06-01T03:31:04.5674457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-01T03:31:04.5686793Z Entering 'third_party/NNPACK' 2024-06-01T03:31:04.5716492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-01T03:31:04.5728771Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T03:31:04.5761187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-01T03:31:04.5774034Z Entering 'third_party/XNNPACK' 2024-06-01T03:31:04.5804828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-01T03:31:04.5827140Z Entering 'third_party/benchmark' 2024-06-01T03:31:04.5860981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-01T03:31:04.5874619Z Entering 'third_party/cpp-httplib' 2024-06-01T03:31:04.5905923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-01T03:31:04.5919425Z Entering 'third_party/cpuinfo' 2024-06-01T03:31:04.5950775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-01T03:31:04.5965477Z Entering 'third_party/cudnn_frontend' 2024-06-01T03:31:04.5996371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-01T03:31:04.6009581Z Entering 'third_party/cutlass' 2024-06-01T03:31:04.6039891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-01T03:31:04.6058595Z Entering 'third_party/eigen' 2024-06-01T03:31:04.6090113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-01T03:31:04.6103322Z Entering 'third_party/fbgemm' 2024-06-01T03:31:04.6134781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-01T03:31:04.6147600Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:31:04.6179304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-01T03:31:04.6192285Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:31:04.6223331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-01T03:31:04.6236497Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:31:04.6266156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-01T03:31:04.6283457Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:31:04.6313556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.6326286Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:31:04.6356840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-01T03:31:04.6369633Z Entering 'third_party/flatbuffers' 2024-06-01T03:31:04.6401667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-01T03:31:04.6415944Z Entering 'third_party/fmt' 2024-06-01T03:31:04.6445816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-01T03:31:04.6459897Z Entering 'third_party/foxi' 2024-06-01T03:31:04.6491520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-01T03:31:04.6502974Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:31:04.6533607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-01T03:31:04.6547050Z Entering 'third_party/gloo' 2024-06-01T03:31:04.6579577Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-01T03:31:04.6592079Z Entering 'third_party/googletest' 2024-06-01T03:31:04.6621912Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.6634416Z Entering 'third_party/ideep' 2024-06-01T03:31:04.6665088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-01T03:31:04.6676947Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T03:31:04.6707227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-01T03:31:04.6725657Z Entering 'third_party/ios-cmake' 2024-06-01T03:31:04.6757289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-06-01T03:31:04.6769756Z Entering 'third_party/ittapi' 2024-06-01T03:31:04.6800374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-01T03:31:04.6814098Z Entering 'third_party/kineto' 2024-06-01T03:31:04.6844686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-01T03:31:04.6859142Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:31:04.6890733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-01T03:31:04.6902941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:31:04.6935128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-01T03:31:04.6949380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:31:04.6981077Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-01T03:31:04.6993849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:31:04.7024781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-01T03:31:04.7038149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:31:04.7069601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-01T03:31:04.7081798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:31:04.7112249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-01T03:31:04.7125882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:31:04.7157352Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-01T03:31:04.7169590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:31:04.7203949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.7216161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:31:04.7246786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-01T03:31:04.7259804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:31:04.7290567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-01T03:31:04.7304642Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:31:04.7335043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-01T03:31:04.7347633Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:31:04.7380123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.7394233Z Entering 'third_party/mimalloc' 2024-06-01T03:31:04.7425459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-01T03:31:04.7437687Z Entering 'third_party/nccl/nccl' 2024-06-01T03:31:04.7469904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-01T03:31:04.7484175Z Entering 'third_party/nlohmann' 2024-06-01T03:31:04.7515415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-01T03:31:04.7528493Z Entering 'third_party/onnx' 2024-06-01T03:31:04.7559148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-01T03:31:04.7583396Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T03:31:04.7614479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-01T03:31:04.7627226Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T03:31:04.7659122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-01T03:31:04.7672723Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T03:31:04.7702925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-01T03:31:04.7715541Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:31:04.7745719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-01T03:31:04.7758307Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:31:04.7788949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.7801675Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:31:04.7833145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-01T03:31:04.7845968Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:31:04.7877241Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-01T03:31:04.7891512Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:31:04.7923381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-01T03:31:04.7934028Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:31:04.7964088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-01T03:31:04.7976365Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:31:04.8023300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-01T03:31:04.8024842Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:31:04.8052760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-01T03:31:04.8065994Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:31:04.8097945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-01T03:31:04.8111988Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:31:04.8141698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-01T03:31:04.8169355Z Entering 'third_party/pocketfft' 2024-06-01T03:31:04.8202011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-01T03:31:04.8215172Z Entering 'third_party/protobuf' 2024-06-01T03:31:04.8246359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-01T03:31:04.8260440Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:31:04.8291446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-01T03:31:04.8303761Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T03:31:04.8334350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.8348862Z Entering 'third_party/psimd' 2024-06-01T03:31:04.8380627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-01T03:31:04.8392564Z Entering 'third_party/pthreadpool' 2024-06-01T03:31:04.8423416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-01T03:31:04.8436302Z Entering 'third_party/pybind11' 2024-06-01T03:31:04.8467199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-01T03:31:04.8479317Z Entering 'third_party/python-peachpy' 2024-06-01T03:31:04.8512129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-01T03:31:04.8525073Z Entering 'third_party/sleef' 2024-06-01T03:31:04.8557394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-01T03:31:04.8571403Z Entering 'third_party/tensorpipe' 2024-06-01T03:31:04.8603152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-01T03:31:04.8615265Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:04.8645295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-01T03:31:04.8657375Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:04.8687599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-01T03:31:04.8700430Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:04.8732991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-01T03:31:04.8743882Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:04.8774879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-01T03:31:04.8785854Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:04.8818159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-01T03:31:04.9765836Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-01T03:31:04.9968503Z Entering 'android/libs/fbjni' 2024-06-01T03:31:04.9995189Z Entering 'third_party/FP16' 2024-06-01T03:31:05.0021189Z Entering 'third_party/FXdiv' 2024-06-01T03:31:05.0047360Z Entering 'third_party/NNPACK' 2024-06-01T03:31:05.0074773Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T03:31:05.0101197Z Entering 'third_party/XNNPACK' 2024-06-01T03:31:05.0139017Z Entering 'third_party/benchmark' 2024-06-01T03:31:05.0165635Z Entering 'third_party/cpp-httplib' 2024-06-01T03:31:05.0192084Z Entering 'third_party/cpuinfo' 2024-06-01T03:31:05.0220237Z Entering 'third_party/cudnn_frontend' 2024-06-01T03:31:05.0249821Z Entering 'third_party/cutlass' 2024-06-01T03:31:05.0281932Z Entering 'third_party/eigen' 2024-06-01T03:31:05.0311701Z Entering 'third_party/fbgemm' 2024-06-01T03:31:05.0339776Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:31:05.0366140Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:31:05.0392707Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:31:05.0422481Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:31:05.0447966Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:31:05.0475959Z Entering 'third_party/flatbuffers' 2024-06-01T03:31:05.0505987Z Entering 'third_party/fmt' 2024-06-01T03:31:05.0535328Z Entering 'third_party/foxi' 2024-06-01T03:31:05.0562910Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:31:05.0590685Z Entering 'third_party/gloo' 2024-06-01T03:31:05.0618944Z Entering 'third_party/googletest' 2024-06-01T03:31:05.0645871Z Entering 'third_party/ideep' 2024-06-01T03:31:05.0674402Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T03:31:05.0707796Z Entering 'third_party/ios-cmake' 2024-06-01T03:31:05.0735740Z Entering 'third_party/ittapi' 2024-06-01T03:31:05.0792408Z Entering 'third_party/kineto' 2024-06-01T03:31:05.0818196Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:31:05.0845246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:31:05.0873071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:31:05.0899630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:31:05.0925041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:31:05.0949793Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:31:05.0979358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:31:05.1007934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:31:05.1033042Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:31:05.1060026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:31:05.1089920Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:31:05.1116713Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:31:05.1141943Z Entering 'third_party/mimalloc' 2024-06-01T03:31:05.1169906Z Entering 'third_party/nccl/nccl' 2024-06-01T03:31:05.1195448Z Entering 'third_party/nlohmann' 2024-06-01T03:31:05.1223943Z Entering 'third_party/onnx' 2024-06-01T03:31:05.1261298Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T03:31:05.1289188Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T03:31:05.1318137Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T03:31:05.1347326Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:31:05.1373791Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:31:05.1398470Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:31:05.1425464Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:31:05.1451021Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:31:05.1475776Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:31:05.1502166Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:31:05.1526949Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:31:05.1553088Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:31:05.1581823Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:31:05.1624518Z Entering 'third_party/pocketfft' 2024-06-01T03:31:05.1651516Z Entering 'third_party/protobuf' 2024-06-01T03:31:05.1679365Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:31:05.1706025Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T03:31:05.1735940Z Entering 'third_party/psimd' 2024-06-01T03:31:05.1764328Z Entering 'third_party/pthreadpool' 2024-06-01T03:31:05.1791530Z Entering 'third_party/pybind11' 2024-06-01T03:31:05.1818807Z Entering 'third_party/python-peachpy' 2024-06-01T03:31:05.1846468Z Entering 'third_party/sleef' 2024-06-01T03:31:05.1874120Z Entering 'third_party/tensorpipe' 2024-06-01T03:31:05.1903423Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:05.1928951Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:05.1955803Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:05.1982644Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:05.2008059Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:05.2046538Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-01T03:31:05.2248211Z Entering 'android/libs/fbjni' 2024-06-01T03:31:05.2275690Z Entering 'third_party/FP16' 2024-06-01T03:31:05.2302503Z Entering 'third_party/FXdiv' 2024-06-01T03:31:05.2328908Z Entering 'third_party/NNPACK' 2024-06-01T03:31:05.2356392Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T03:31:05.2382618Z Entering 'third_party/XNNPACK' 2024-06-01T03:31:05.2421163Z Entering 'third_party/benchmark' 2024-06-01T03:31:05.2449016Z Entering 'third_party/cpp-httplib' 2024-06-01T03:31:05.2476906Z Entering 'third_party/cpuinfo' 2024-06-01T03:31:05.2505716Z Entering 'third_party/cudnn_frontend' 2024-06-01T03:31:05.2534114Z Entering 'third_party/cutlass' 2024-06-01T03:31:05.2565860Z Entering 'third_party/eigen' 2024-06-01T03:31:05.2595367Z Entering 'third_party/fbgemm' 2024-06-01T03:31:05.2622342Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T03:31:05.2648822Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T03:31:05.2675441Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T03:31:05.2707698Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T03:31:05.2736197Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T03:31:05.2765787Z Entering 'third_party/flatbuffers' 2024-06-01T03:31:05.2794768Z Entering 'third_party/fmt' 2024-06-01T03:31:05.2822051Z Entering 'third_party/foxi' 2024-06-01T03:31:05.2848286Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T03:31:05.2876199Z Entering 'third_party/gloo' 2024-06-01T03:31:05.2905309Z Entering 'third_party/googletest' 2024-06-01T03:31:05.2931902Z Entering 'third_party/ideep' 2024-06-01T03:31:05.2957555Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T03:31:05.2989824Z Entering 'third_party/ios-cmake' 2024-06-01T03:31:05.3018224Z Entering 'third_party/ittapi' 2024-06-01T03:31:05.3046567Z Entering 'third_party/kineto' 2024-06-01T03:31:05.3074752Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T03:31:05.3102159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T03:31:05.3129984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T03:31:05.3155404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T03:31:05.3181454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T03:31:05.3208203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T03:31:05.3236810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T03:31:05.3263208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T03:31:05.3290268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T03:31:05.3318351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T03:31:05.3346323Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T03:31:05.3371656Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T03:31:05.3400873Z Entering 'third_party/mimalloc' 2024-06-01T03:31:05.3428182Z Entering 'third_party/nccl/nccl' 2024-06-01T03:31:05.3457159Z Entering 'third_party/nlohmann' 2024-06-01T03:31:05.3485376Z Entering 'third_party/onnx' 2024-06-01T03:31:05.3521511Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T03:31:05.3549372Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T03:31:05.3581516Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T03:31:05.3609735Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T03:31:05.3637970Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T03:31:05.3666470Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T03:31:05.3693852Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T03:31:05.3720653Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T03:31:05.3748546Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T03:31:05.3776642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T03:31:05.3801704Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T03:31:05.3828596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T03:31:05.3858664Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T03:31:05.3899523Z Entering 'third_party/pocketfft' 2024-06-01T03:31:05.3927242Z Entering 'third_party/protobuf' 2024-06-01T03:31:05.3955797Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T03:31:05.3982718Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T03:31:05.4010145Z Entering 'third_party/psimd' 2024-06-01T03:31:05.4036882Z Entering 'third_party/pthreadpool' 2024-06-01T03:31:05.4062942Z Entering 'third_party/pybind11' 2024-06-01T03:31:05.4093234Z Entering 'third_party/python-peachpy' 2024-06-01T03:31:05.4120572Z Entering 'third_party/sleef' 2024-06-01T03:31:05.4149577Z Entering 'third_party/tensorpipe' 2024-06-01T03:31:05.4179111Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T03:31:05.4205345Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T03:31:05.4232450Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T03:31:05.4257291Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T03:31:05.4283463Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T03:31:05.4321531Z ##[endgroup] 2024-06-01T03:31:05.4349438Z [command]/usr/bin/git log -1 --format='%H' 2024-06-01T03:31:05.4369384Z 'de352ff31081bc3b80baf4f72168a00bdf6cccae' 2024-06-01T03:31:05.4548014Z Prepare all required actions 2024-06-01T03:31:05.4548464Z Getting action download info 2024-06-01T03:31:05.5740211Z ##[group]Run ./.github/actions/setup-linux 2024-06-01T03:31:05.5740654Z env: 2024-06-01T03:31:05.5740942Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:05.5741298Z ##[endgroup] 2024-06-01T03:31:05.5800659Z ##[group]Run set -euo pipefail 2024-06-01T03:31:05.5801093Z set -euo pipefail 2024-06-01T03:31:05.5801499Z function get_ec2_metadata() { 2024-06-01T03:31:05.5802055Z  # Pulled from instance metadata endpoint for EC2 2024-06-01T03:31:05.5803063Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-01T03:31:05.5803850Z  category=$1 2024-06-01T03:31:05.5804392Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-01T03:31:05.5805041Z  runner_name_str=i-0b128c63f91218fe4 2024-06-01T03:31:05.5805551Z  if [[ -f /.inarc ]]; then 2024-06-01T03:31:05.5806059Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-01T03:31:05.5806635Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-01T03:31:05.5807346Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-01T03:31:05.5807967Z  else 2024-06-01T03:31:05.5808473Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-01T03:31:05.5809057Z  fi 2024-06-01T03:31:05.5809459Z } 2024-06-01T03:31:05.5809829Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-01T03:31:05.5810438Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-01T03:31:05.5811112Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-01T03:31:05.5811702Z echo "system info $(uname -a)" 2024-06-01T03:31:05.5819375Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:05.5819900Z env: 2024-06-01T03:31:05.5820184Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:05.5820547Z ##[endgroup] 2024-06-01T03:31:05.5890986Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-01T03:31:05.5932259Z instance-id: i-0b128c63f91218fe4 2024-06-01T03:31:05.5981094Z instance-type: g5.4xlarge 2024-06-01T03:31:05.5987037Z system info Linux ip-10-0-52-128.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-01T03:31:05.6007109Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-01T03:31:05.6008071Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-01T03:31:05.6015757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:05.6016268Z env: 2024-06-01T03:31:05.6016553Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:05.6016906Z ##[endgroup] 2024-06-01T03:31:05.6094700Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-01T03:31:05.6095342Z if systemctl is-active --quiet docker; then 2024-06-01T03:31:05.6095898Z  echo "Docker daemon is running..."; 2024-06-01T03:31:05.6096358Z else 2024-06-01T03:31:05.6096861Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-01T03:31:05.6097465Z fi 2024-06-01T03:31:05.6104677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:05.6105192Z env: 2024-06-01T03:31:05.6105468Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:05.6105833Z ##[endgroup] 2024-06-01T03:31:05.6142485Z Docker daemon is running... 2024-06-01T03:31:05.6186200Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-01T03:31:05.6186783Z with: 2024-06-01T03:31:05.6187054Z shell: bash 2024-06-01T03:31:05.6187349Z timeout_minutes: 5 2024-06-01T03:31:05.6187683Z max_attempts: 3 2024-06-01T03:31:05.6188007Z retry_wait_seconds: 30 2024-06-01T03:31:05.6189565Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-01T03:31:05.6191271Z polling_interval_seconds: 1 2024-06-01T03:31:05.6191663Z warning_on_retry: true 2024-06-01T03:31:05.6192023Z continue_on_error: false 2024-06-01T03:31:05.6192376Z env: 2024-06-01T03:31:05.6192645Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:05.6193011Z AWS_RETRY_MODE: standard 2024-06-01T03:31:05.6193379Z AWS_MAX_ATTEMPTS: 5 2024-06-01T03:31:05.6193721Z AWS_DEFAULT_REGION: us-east-1 2024-06-01T03:31:05.6194103Z ##[endgroup] 2024-06-01T03:31:06.4653952Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-01T03:31:06.4655084Z Configure a credential helper to remove this warning. See 2024-06-01T03:31:06.4656100Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-01T03:31:06.4656634Z 2024-06-01T03:31:06.4656787Z Login Succeeded 2024-06-01T03:31:06.6678799Z Command completed after 1 attempt(s). 2024-06-01T03:31:06.6721112Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-01T03:31:06.6721862Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-01T03:31:06.6722645Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-01T03:31:06.6730204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:06.6730721Z env: 2024-06-01T03:31:06.6731113Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.6731459Z ##[endgroup] 2024-06-01T03:31:06.6787420Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-01T03:31:06.6788226Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-01T03:31:06.6788841Z # shellcheck disable=SC2046 2024-06-01T03:31:06.6789317Z docker stop $(docker ps -q) || true 2024-06-01T03:31:06.6789829Z # Prune all of the docker images 2024-06-01T03:31:06.6790678Z docker system prune -af 2024-06-01T03:31:06.6797490Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:06.6798065Z env: 2024-06-01T03:31:06.6798353Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.6798704Z ##[endgroup] 2024-06-01T03:31:06.7103939Z "docker stop" requires at least 1 argument. 2024-06-01T03:31:06.7104618Z See 'docker stop --help'. 2024-06-01T03:31:06.7104868Z 2024-06-01T03:31:06.7105101Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-01T03:31:06.7105526Z 2024-06-01T03:31:06.7105678Z Stop one or more running containers 2024-06-01T03:31:06.7303001Z Total reclaimed space: 0B 2024-06-01T03:31:06.7334498Z ##[group]Run set +e 2024-06-01T03:31:06.7334870Z set +e 2024-06-01T03:31:06.7335183Z set -x 2024-06-01T03:31:06.7335485Z  2024-06-01T03:31:06.7335819Z PT_DOMAIN=download.pytorch.org 2024-06-01T03:31:06.7336631Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-01T03:31:06.7337731Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-01T03:31:06.7338512Z # one is returned at random 2024-06-01T03:31:06.7339075Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-01T03:31:06.7339602Z  2024-06-01T03:31:06.7339923Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-01T03:31:06.7340559Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-01T03:31:06.7341351Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-01T03:31:06.7341926Z  2024-06-01T03:31:06.7342248Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-01T03:31:06.7342810Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-01T03:31:06.7343331Z  exit 1 2024-06-01T03:31:06.7343650Z  fi 2024-06-01T03:31:06.7343936Z fi 2024-06-01T03:31:06.7344327Z  2024-06-01T03:31:06.7344703Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-01T03:31:06.7345228Z  # Clean up any old records first 2024-06-01T03:31:06.7345750Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-01T03:31:06.7346216Z fi 2024-06-01T03:31:06.7346505Z  2024-06-01T03:31:06.7346938Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-01T03:31:06.7347494Z cat /etc/hosts 2024-06-01T03:31:06.7355337Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:06.7355857Z env: 2024-06-01T03:31:06.7356131Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.7356489Z ##[endgroup] 2024-06-01T03:31:06.7376781Z + PT_DOMAIN=download.pytorch.org 2024-06-01T03:31:06.7380627Z ++ dig -4 +short download.pytorch.org 2024-06-01T03:31:06.7381081Z ++ tail -n1 2024-06-01T03:31:06.7486037Z + RESOLVED_IP=18.160.10.36 2024-06-01T03:31:06.7486479Z + '[' -z 18.160.10.36 ']' 2024-06-01T03:31:06.7486923Z + grep -r download.pytorch.org /etc/hosts 2024-06-01T03:31:06.7493129Z + echo '18.160.10.36 download.pytorch.org' 2024-06-01T03:31:06.7493770Z + sudo tee -a /etc/hosts 2024-06-01T03:31:06.7571674Z 18.160.10.36 download.pytorch.org 2024-06-01T03:31:06.7585302Z + cat /etc/hosts 2024-06-01T03:31:06.7589890Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-01T03:31:06.7602635Z ::1 localhost6 localhost6.localdomain6 2024-06-01T03:31:06.7603139Z 18.160.10.36 download.pytorch.org 2024-06-01T03:31:06.7757280Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-01T03:31:06.7757912Z with: 2024-06-01T03:31:06.7759035Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7760281Z docker-build-dir: .ci/docker 2024-06-01T03:31:06.7760671Z working-directory: . 2024-06-01T03:31:06.7761151Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:06.7761691Z force-push: false 2024-06-01T03:31:06.7761995Z env: 2024-06-01T03:31:06.7762398Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.7762767Z ##[endgroup] 2024-06-01T03:31:06.7779767Z ##[group]Run set -ex 2024-06-01T03:31:06.7780141Z set -ex 2024-06-01T03:31:06.7780492Z  2024-06-01T03:31:06.7781126Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-01T03:31:06.7782146Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-01T03:31:06.7782967Z # job could then download the pre-built image as usual 2024-06-01T03:31:06.7783725Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-01T03:31:06.7784413Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7785067Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7785638Z  2024-06-01T03:31:06.7786170Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-01T03:31:06.7786819Z  exit 0 2024-06-01T03:31:06.7787131Z else 2024-06-01T03:31:06.7787501Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7787965Z fi 2024-06-01T03:31:06.7788251Z  2024-06-01T03:31:06.7788737Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-01T03:31:06.7789609Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-01T03:31:06.7790605Z  # use it as it is, but first let's extract the tag 2024-06-01T03:31:06.7791335Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-01T03:31:06.7792088Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7792810Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7793384Z else 2024-06-01T03:31:06.7793833Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-01T03:31:06.7794518Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7795452Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7796224Z fi 2024-06-01T03:31:06.7803567Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:06.7804076Z env: 2024-06-01T03:31:06.7804360Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.7804716Z REPO_NAME: pytorch 2024-06-01T03:31:06.7805876Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7807112Z DOCKER_BUILD_DIR: .ci/docker 2024-06-01T03:31:06.7807621Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:06.7808142Z ##[endgroup] 2024-06-01T03:31:06.7827646Z + [[ ! -d .ci/docker ]] 2024-06-01T03:31:06.7828177Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-01T03:31:06.7828575Z + echo skip=false 2024-06-01T03:31:06.7830776Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-01T03:31:06.7833478Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7834711Z ++ awk -F '[:,]' '{print $2}' 2024-06-01T03:31:06.7838744Z + DOCKER_TAG=7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7839447Z + echo docker-tag=7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7841193Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7865873Z ##[group]Run set +e 2024-06-01T03:31:06.7866229Z set +e 2024-06-01T03:31:06.7866530Z set -x 2024-06-01T03:31:06.7866832Z  2024-06-01T03:31:06.7867103Z login() { 2024-06-01T03:31:06.7867782Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-01T03:31:06.7868516Z } 2024-06-01T03:31:06.7868794Z  2024-06-01T03:31:06.7869065Z retry () { 2024-06-01T03:31:06.7869459Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-01T03:31:06.7869915Z } 2024-06-01T03:31:06.7870430Z  2024-06-01T03:31:06.7870749Z retry login "${DOCKER_REGISTRY}" 2024-06-01T03:31:06.7871181Z  2024-06-01T03:31:06.7871670Z # Check if image already exists, if it does then skip building it 2024-06-01T03:31:06.7872403Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-01T03:31:06.7872924Z  exit 0 2024-06-01T03:31:06.7873231Z fi 2024-06-01T03:31:06.7873514Z  2024-06-01T03:31:06.7874025Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-01T03:31:06.7874879Z # be empty. The default action would be to continue rebuild the image 2024-06-01T03:31:06.7875645Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-01T03:31:06.7876345Z  # if we're on the base branch then use the parent commit 2024-06-01T03:31:06.7876956Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-01T03:31:06.7877391Z else 2024-06-01T03:31:06.7877870Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-01T03:31:06.7878628Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-01T03:31:06.7879151Z fi 2024-06-01T03:31:06.7879429Z  2024-06-01T03:31:06.7879756Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-01T03:31:06.7880277Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7880739Z  2024-06-01T03:31:06.7881418Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-01T03:31:06.7882206Z  exit 0 2024-06-01T03:31:06.7882626Z fi 2024-06-01T03:31:06.7882908Z  2024-06-01T03:31:06.7883360Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-01T03:31:06.7884367Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-01T03:31:06.7885198Z  exit 1 2024-06-01T03:31:06.7885502Z fi 2024-06-01T03:31:06.7885786Z  2024-06-01T03:31:06.7886290Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-01T03:31:06.7887275Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-01T03:31:06.7888170Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-01T03:31:06.7889160Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-01T03:31:06.7890307Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-01T03:31:06.7891150Z fi 2024-06-01T03:31:06.7891439Z  2024-06-01T03:31:06.7891795Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-01T03:31:06.7898460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:06.7898978Z env: 2024-06-01T03:31:06.7899257Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:06.7899620Z DOCKER_BUILD_DIR: .ci/docker 2024-06-01T03:31:06.7900103Z BASE_REVISION: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:31:06.7901428Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7902723Z DOCKER_TAG: 7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:06.7903330Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:06.7903859Z ##[endgroup] 2024-06-01T03:31:06.7923924Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:06.7924626Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:06.7925479Z + aws ecr get-login-password --region us-east-1 2024-06-01T03:31:06.7926294Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.2242847Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-01T03:31:07.2243716Z Configure a credential helper to remove this warning. See 2024-06-01T03:31:07.2244567Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-01T03:31:07.2245109Z 2024-06-01T03:31:07.2245231Z Login Succeeded 2024-06-01T03:31:07.2256751Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.4334487Z { 2024-06-01T03:31:07.4335107Z "schemaVersion": 2, 2024-06-01T03:31:07.4336120Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-01T03:31:07.4336954Z "config": { 2024-06-01T03:31:07.4337815Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-01T03:31:07.4338827Z "size": 45564, 2024-06-01T03:31:07.4339823Z "digest": "sha256:d37233509623aba91eac13ad223e1b9eed85dd9651dcb816d1c4217e406e176c" 2024-06-01T03:31:07.4340955Z }, 2024-06-01T03:31:07.4341300Z "layers": [ 2024-06-01T03:31:07.4341664Z { 2024-06-01T03:31:07.4342289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4343236Z "size": 28584317, 2024-06-01T03:31:07.4344073Z "digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05" 2024-06-01T03:31:07.4345087Z }, 2024-06-01T03:31:07.4345464Z { 2024-06-01T03:31:07.4346192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4347194Z "size": 7944771, 2024-06-01T03:31:07.4348269Z "digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6" 2024-06-01T03:31:07.4349218Z }, 2024-06-01T03:31:07.4349578Z { 2024-06-01T03:31:07.4350551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4351330Z "size": 57593718, 2024-06-01T03:31:07.4352142Z "digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481" 2024-06-01T03:31:07.4352968Z }, 2024-06-01T03:31:07.4353664Z { 2024-06-01T03:31:07.4354482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4355392Z "size": 187, 2024-06-01T03:31:07.4356119Z "digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6" 2024-06-01T03:31:07.4356925Z }, 2024-06-01T03:31:07.4357241Z { 2024-06-01T03:31:07.4357797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4358519Z "size": 6885, 2024-06-01T03:31:07.4359075Z "digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488" 2024-06-01T03:31:07.4359692Z }, 2024-06-01T03:31:07.4360200Z { 2024-06-01T03:31:07.4360646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4361219Z "size": 1361380219, 2024-06-01T03:31:07.4361787Z "digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46" 2024-06-01T03:31:07.4362599Z }, 2024-06-01T03:31:07.4362997Z + exit 0 2024-06-01T03:31:07.4363256Z { 2024-06-01T03:31:07.4363687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4364258Z "size": 62686, 2024-06-01T03:31:07.4364820Z "digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3" 2024-06-01T03:31:07.4365446Z }, 2024-06-01T03:31:07.4365696Z { 2024-06-01T03:31:07.4366132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4366697Z "size": 1685, 2024-06-01T03:31:07.4367236Z "digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9" 2024-06-01T03:31:07.4367873Z }, 2024-06-01T03:31:07.4368132Z { 2024-06-01T03:31:07.4368569Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4369135Z "size": 1523, 2024-06-01T03:31:07.4369685Z "digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d" 2024-06-01T03:31:07.4370317Z }, 2024-06-01T03:31:07.4370562Z { 2024-06-01T03:31:07.4370999Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4371572Z "size": 2608186849, 2024-06-01T03:31:07.4372162Z "digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8" 2024-06-01T03:31:07.4372794Z }, 2024-06-01T03:31:07.4373043Z { 2024-06-01T03:31:07.4373482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4374046Z "size": 86631, 2024-06-01T03:31:07.4374605Z "digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab" 2024-06-01T03:31:07.4375242Z }, 2024-06-01T03:31:07.4375488Z { 2024-06-01T03:31:07.4391923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4392552Z "size": 1822, 2024-06-01T03:31:07.4393122Z "digest": "sha256:fec42e0e4ca28598126d018fae7d2bbf7ab2f4201332c748797d4cfc09d2e208" 2024-06-01T03:31:07.4393766Z }, 2024-06-01T03:31:07.4394015Z { 2024-06-01T03:31:07.4394464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4395041Z "size": 245768134, 2024-06-01T03:31:07.4395622Z "digest": "sha256:d0d1bc0363fb6f875c026d483df0199ec24dc2628c35aa572a490deeb6101534" 2024-06-01T03:31:07.4396253Z }, 2024-06-01T03:31:07.4396496Z { 2024-06-01T03:31:07.4396941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4397503Z "size": 545, 2024-06-01T03:31:07.4398050Z "digest": "sha256:cc64afd638a7c666257b7007f053d414f1a0e9c7920c1c68c258925f4364fb75" 2024-06-01T03:31:07.4398677Z }, 2024-06-01T03:31:07.4398922Z { 2024-06-01T03:31:07.4399370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4399946Z "size": 1277, 2024-06-01T03:31:07.4400492Z "digest": "sha256:52f0d9fdf39671a46c1f42af7847772434965ccb3a9d09b8e850bf8db0256079" 2024-06-01T03:31:07.4401126Z }, 2024-06-01T03:31:07.4401377Z { 2024-06-01T03:31:07.4401807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4402475Z "size": 482, 2024-06-01T03:31:07.4403020Z "digest": "sha256:65a297b68fd13f7d3d7e1508cc28112a0701e3fae333bbb3fd73061ae4ae1878" 2024-06-01T03:31:07.4403650Z }, 2024-06-01T03:31:07.4403890Z { 2024-06-01T03:31:07.4404335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4404904Z "size": 91711146, 2024-06-01T03:31:07.4405480Z "digest": "sha256:424e127052d0af9db673bdb5dc57af7fb8fe272df1107d0eca7c676e8b5e54e8" 2024-06-01T03:31:07.4406110Z }, 2024-06-01T03:31:07.4406354Z { 2024-06-01T03:31:07.4406791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4407592Z "size": 3241, 2024-06-01T03:31:07.4408203Z "digest": "sha256:184e6daf3e53c447f00c6f8a9951fb5dc29d2a9a00953a3cb5f52cb9d77aa9b7" 2024-06-01T03:31:07.4408844Z }, 2024-06-01T03:31:07.4409092Z { 2024-06-01T03:31:07.4409526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4410093Z "size": 1904, 2024-06-01T03:31:07.4410641Z "digest": "sha256:7a163619b00355f62e59f312d1795cca3fe08afeb48f2884c6047a574f313959" 2024-06-01T03:31:07.4411277Z }, 2024-06-01T03:31:07.4411518Z { 2024-06-01T03:31:07.4411954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4412517Z "size": 699, 2024-06-01T03:31:07.4413067Z "digest": "sha256:2f1d5eae98e57c4aa4ee8f575c82438ccc03a583d5a62d821acfcee4d4fbd65d" 2024-06-01T03:31:07.4413693Z }, 2024-06-01T03:31:07.4413937Z { 2024-06-01T03:31:07.4414380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4414952Z "size": 2797577861, 2024-06-01T03:31:07.4415537Z "digest": "sha256:22ed7fbe39a23341d1e7f50dd4e20e46228acc11e5fe52d6cdc24a84006a39e5" 2024-06-01T03:31:07.4416176Z }, 2024-06-01T03:31:07.4416424Z { 2024-06-01T03:31:07.4416855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4417416Z "size": 379, 2024-06-01T03:31:07.4418000Z "digest": "sha256:89b2bc3b82621ef76a4ffba0a5573de7626c999a70d6ed54beede1fea5a42f85" 2024-06-01T03:31:07.4418664Z }, 2024-06-01T03:31:07.4418904Z { 2024-06-01T03:31:07.4419341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4419904Z "size": 11879, 2024-06-01T03:31:07.4420460Z "digest": "sha256:fc61ad516d94690b9d3703d3dae5e0735519b96ba6aa606db830a0c3cef581f1" 2024-06-01T03:31:07.4421082Z }, 2024-06-01T03:31:07.4421328Z { 2024-06-01T03:31:07.4421766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4422320Z "size": 802, 2024-06-01T03:31:07.4422886Z "digest": "sha256:1aafe81a9a9a8770f689b785c3cb34d940c90ebd5a45dc0a50be0aab029fabca" 2024-06-01T03:31:07.4423534Z }, 2024-06-01T03:31:07.4423781Z { 2024-06-01T03:31:07.4424216Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4424786Z "size": 106, 2024-06-01T03:31:07.4425341Z "digest": "sha256:9087b5d4aba087cdedb4c68156112a414e553b7a56d5ae2a3e6f70e02f7e9f86" 2024-06-01T03:31:07.4425977Z }, 2024-06-01T03:31:07.4426218Z { 2024-06-01T03:31:07.4426651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4427224Z "size": 502, 2024-06-01T03:31:07.4427758Z "digest": "sha256:41894f89ccf0d45e26f80654c83680e247c8a69a07e9ba40a695790fc2f2d620" 2024-06-01T03:31:07.4428437Z }, 2024-06-01T03:31:07.4428683Z { 2024-06-01T03:31:07.4429119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4429680Z "size": 121477173, 2024-06-01T03:31:07.4430624Z "digest": "sha256:82ae7e4ef3ff0dde3ace22a8fe0e31f8f2f279479206995c593964fd0b1a01d9" 2024-06-01T03:31:07.4431418Z }, 2024-06-01T03:31:07.4431684Z { 2024-06-01T03:31:07.4432108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4432669Z "size": 109, 2024-06-01T03:31:07.4433201Z "digest": "sha256:080b4f32181666e873644ec8d4459fe524f9249c71eff9b66ab2d895d519e58a" 2024-06-01T03:31:07.4433808Z }, 2024-06-01T03:31:07.4434043Z { 2024-06-01T03:31:07.4434480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4435036Z "size": 488, 2024-06-01T03:31:07.4435582Z "digest": "sha256:9f98bff3bbd53f03e88414ec41ad1d7a1eac80cd5dfc5f8d967faa471bfa3e0d" 2024-06-01T03:31:07.4436213Z }, 2024-06-01T03:31:07.4436455Z { 2024-06-01T03:31:07.4436890Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4437436Z "size": 297, 2024-06-01T03:31:07.4437971Z "digest": "sha256:c8515406c82cc797516565a1b7ff9eae1644b08ee067cae9b0da33ce84e6e45e" 2024-06-01T03:31:07.4438775Z }, 2024-06-01T03:31:07.4439023Z { 2024-06-01T03:31:07.4439446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4440009Z "size": 103, 2024-06-01T03:31:07.4440560Z "digest": "sha256:dba13cb5868ee5e0c5ad4723274d567bd9ddf6b8b39aea44600932cc7eda788a" 2024-06-01T03:31:07.4441199Z }, 2024-06-01T03:31:07.4441434Z { 2024-06-01T03:31:07.4441868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4442519Z "size": 1473, 2024-06-01T03:31:07.4443052Z "digest": "sha256:f2036f97fbab53cdcc47137c130745061d4341c97b22a69516e1d54d8afdbd71" 2024-06-01T03:31:07.4443670Z }, 2024-06-01T03:31:07.4443912Z { 2024-06-01T03:31:07.4444348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4444898Z "size": 446582944, 2024-06-01T03:31:07.4445456Z "digest": "sha256:de273ac1757956c5488a1706f14b45f14d31daf6ccd8b3b2f13b2b42f0a842e9" 2024-06-01T03:31:07.4446091Z }, 2024-06-01T03:31:07.4446347Z { 2024-06-01T03:31:07.4446774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4447324Z "size": 159, 2024-06-01T03:31:07.4447867Z "digest": "sha256:29ca9539f3ae1ae1e820e2a74d96cfe698f86f2eb8b665752882bf6c8d750219" 2024-06-01T03:31:07.4448482Z }, 2024-06-01T03:31:07.4448725Z { 2024-06-01T03:31:07.4449159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4449713Z "size": 562, 2024-06-01T03:31:07.4450249Z "digest": "sha256:9f31f86a8b955161fcbe4116749df303cd7406dd80f8d902effd897de0f87ab5" 2024-06-01T03:31:07.4450871Z }, 2024-06-01T03:31:07.4451116Z { 2024-06-01T03:31:07.4451545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4452090Z "size": 35867238, 2024-06-01T03:31:07.4452853Z "digest": "sha256:d2455d25abe3542ef769aea98ac1c61bbe785789cf157a5c1bf3d83b0b57d764" 2024-06-01T03:31:07.4453498Z }, 2024-06-01T03:31:07.4453738Z { 2024-06-01T03:31:07.4454180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4454738Z "size": 104, 2024-06-01T03:31:07.4455280Z "digest": "sha256:859146fb20cc06754c0c40cdb5b0bb05d0879a793fb6aee3af78f2c861728484" 2024-06-01T03:31:07.4455905Z }, 2024-06-01T03:31:07.4456143Z { 2024-06-01T03:31:07.4456575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4457130Z "size": 424, 2024-06-01T03:31:07.4457675Z "digest": "sha256:d401e21441f66d561c8185bffed360dfc305137a8aebb50e5aa1fa0c48eb55f8" 2024-06-01T03:31:07.4458354Z }, 2024-06-01T03:31:07.4458598Z { 2024-06-01T03:31:07.4459029Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4459584Z "size": 20261941, 2024-06-01T03:31:07.4460137Z "digest": "sha256:f80c9a959f817c2a14940ae04308c37d27040a17a541c1d21082190fe8e6856c" 2024-06-01T03:31:07.4460756Z }, 2024-06-01T03:31:07.4460992Z { 2024-06-01T03:31:07.4461432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4461999Z "size": 439, 2024-06-01T03:31:07.4462535Z "digest": "sha256:5ad81974f1264156ae41773cc872228b9dbb7e6f453fe740319964982112901a" 2024-06-01T03:31:07.4463143Z }, 2024-06-01T03:31:07.4463389Z { 2024-06-01T03:31:07.4463816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4464370Z "size": 699, 2024-06-01T03:31:07.4464911Z "digest": "sha256:2f1d5eae98e57c4aa4ee8f575c82438ccc03a583d5a62d821acfcee4d4fbd65d" 2024-06-01T03:31:07.4465544Z }, 2024-06-01T03:31:07.4465785Z { 2024-06-01T03:31:07.4466205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4466757Z "size": 142, 2024-06-01T03:31:07.4467288Z "digest": "sha256:25829e6ccd208637271cb89b768caf212f003f97d1b57852da821f01b3c9d3ea" 2024-06-01T03:31:07.4467898Z }, 2024-06-01T03:31:07.4468149Z { 2024-06-01T03:31:07.4468628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4469312Z "size": 136, 2024-06-01T03:31:07.4469860Z "digest": "sha256:672438d05e4a354b2cb29f1f22e9b3a8797ad53ec8f443b53415cf7e91ea887c" 2024-06-01T03:31:07.4470647Z }, 2024-06-01T03:31:07.4470897Z { 2024-06-01T03:31:07.4471330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4471909Z "size": 4980825073, 2024-06-01T03:31:07.4472477Z "digest": "sha256:e5eecd30f362109e41643de20e27f83be3a2fef99c3d7de37eb7109de46827de" 2024-06-01T03:31:07.4473103Z }, 2024-06-01T03:31:07.4473349Z { 2024-06-01T03:31:07.4473772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4474330Z "size": 189, 2024-06-01T03:31:07.4474880Z "digest": "sha256:23b679499cb9cd173d0fcd41e08d7c56caa14bd2d8b249b12bd3f94c31c3bbf1" 2024-06-01T03:31:07.4475508Z }, 2024-06-01T03:31:07.4475746Z { 2024-06-01T03:31:07.4476174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4476736Z "size": 562, 2024-06-01T03:31:07.4477287Z "digest": "sha256:e139f208fe9152529a8d5d9fafd9e3549581ab6f50607dcf5ac8fc4c53817970" 2024-06-01T03:31:07.4477926Z }, 2024-06-01T03:31:07.4478200Z { 2024-06-01T03:31:07.4478631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4479187Z "size": 43162742, 2024-06-01T03:31:07.4479758Z "digest": "sha256:4c505cd637773dca0e07d3d1b01bc7adeffa29f0e2db1b20ad8a0fdfb89769c6" 2024-06-01T03:31:07.4480393Z }, 2024-06-01T03:31:07.4480636Z { 2024-06-01T03:31:07.4481057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4481608Z "size": 106, 2024-06-01T03:31:07.4482139Z "digest": "sha256:d432f274bf714c5cf1f1543d91777069304bc990702f6caa684cd4e0eb044a09" 2024-06-01T03:31:07.4482833Z }, 2024-06-01T03:31:07.4483073Z { 2024-06-01T03:31:07.4483501Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4484079Z "size": 1212, 2024-06-01T03:31:07.4484631Z "digest": "sha256:a960cc0a9498899ac3b00f9a23952173c011cda9e4eafa67898b4d8bdb0e7cac" 2024-06-01T03:31:07.4485245Z }, 2024-06-01T03:31:07.4485488Z { 2024-06-01T03:31:07.4485914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4486468Z "size": 699, 2024-06-01T03:31:07.4487020Z "digest": "sha256:2f1d5eae98e57c4aa4ee8f575c82438ccc03a583d5a62d821acfcee4d4fbd65d" 2024-06-01T03:31:07.4487652Z }, 2024-06-01T03:31:07.4487896Z { 2024-06-01T03:31:07.4488324Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4488875Z "size": 136, 2024-06-01T03:31:07.4489417Z "digest": "sha256:333ad8bb3f6ee2781a4cc066c4f1b4510173501a47ceb30eee27f4fa159cecc8" 2024-06-01T03:31:07.4490041Z }, 2024-06-01T03:31:07.4490284Z { 2024-06-01T03:31:07.4490713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4491273Z "size": 119, 2024-06-01T03:31:07.4491817Z "digest": "sha256:d5d8665989edb4af9053537e41cf2354e4754084fe4456c8c918ebbec0cf6825" 2024-06-01T03:31:07.4492429Z }, 2024-06-01T03:31:07.4492680Z { 2024-06-01T03:31:07.4493115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4493671Z "size": 1855122882, 2024-06-01T03:31:07.4494240Z "digest": "sha256:a5d8e3034f5666b0a64421b24aed59a960061b80c4e5fd372bc2833fc647232d" 2024-06-01T03:31:07.4494859Z }, 2024-06-01T03:31:07.4495107Z { 2024-06-01T03:31:07.4495534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4496100Z "size": 171, 2024-06-01T03:31:07.4496625Z "digest": "sha256:292a840d61927c71316fbcdf0777752248e311308ccbf918d7d49c2976793f07" 2024-06-01T03:31:07.4497244Z }, 2024-06-01T03:31:07.4497481Z { 2024-06-01T03:31:07.4497932Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4498536Z "size": 1841, 2024-06-01T03:31:07.4499074Z "digest": "sha256:545ad07eb1b0530edd88db3019cd3b58a7f2b214772133b6512247427a55c1e8" 2024-06-01T03:31:07.4499861Z }, 2024-06-01T03:31:07.4500111Z { 2024-06-01T03:31:07.4500548Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4501113Z "size": 7529786, 2024-06-01T03:31:07.4501666Z "digest": "sha256:69b414c0796e9e531ec92350e4922ae8e8e2db81ea69a9d32ef866b096f5f53e" 2024-06-01T03:31:07.4502296Z }, 2024-06-01T03:31:07.4502541Z { 2024-06-01T03:31:07.4502969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4503543Z "size": 163, 2024-06-01T03:31:07.4504086Z "digest": "sha256:20566ec9e8193d3074d6e6e91c4455ecf517ce1b4e4304d876b2d4a44ebf79b0" 2024-06-01T03:31:07.4504708Z }, 2024-06-01T03:31:07.4504944Z { 2024-06-01T03:31:07.4505383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4505945Z "size": 7939, 2024-06-01T03:31:07.4506496Z "digest": "sha256:dfba59f41e5c1deae00978ae519c83800cd3ceb6eb1933645dbed27f7788debb" 2024-06-01T03:31:07.4507128Z }, 2024-06-01T03:31:07.4507385Z { 2024-06-01T03:31:07.4507818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4508375Z "size": 8069, 2024-06-01T03:31:07.4508930Z "digest": "sha256:ad96f96c6a612bb4be0c7334499102784fedb1d20b530efd478a3cfcfb3e9cb8" 2024-06-01T03:31:07.4509579Z }, 2024-06-01T03:31:07.4509825Z { 2024-06-01T03:31:07.4510428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4510987Z "size": 302, 2024-06-01T03:31:07.4511525Z "digest": "sha256:3064c9882e6e1af2169da852b7194fe5c3b39fa8aedb3047f3cb994f50d4b867" 2024-06-01T03:31:07.4512147Z }, 2024-06-01T03:31:07.4512381Z { 2024-06-01T03:31:07.4512814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4513366Z "size": 7628726, 2024-06-01T03:31:07.4513914Z "digest": "sha256:eb8bda3e2c438b40c19f9b81dca657f1098195e0f1c967e0062dc5b1ccd09ad0" 2024-06-01T03:31:07.4514539Z }, 2024-06-01T03:31:07.4514783Z { 2024-06-01T03:31:07.4515225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4515775Z "size": 108, 2024-06-01T03:31:07.4516303Z "digest": "sha256:05ed995317dab2003a6ffa54b98d49912ec4e7029405459d1377589d5bc7abfd" 2024-06-01T03:31:07.4516911Z }, 2024-06-01T03:31:07.4517151Z { 2024-06-01T03:31:07.4517571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4518121Z "size": 54145477, 2024-06-01T03:31:07.4518668Z "digest": "sha256:178627ca8d561070a7f821445b8ce92e2e82a5ce73a8526d5166e988ab8f6e63" 2024-06-01T03:31:07.4519268Z }, 2024-06-01T03:31:07.4519511Z { 2024-06-01T03:31:07.4519942Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4520494Z "size": 507, 2024-06-01T03:31:07.4521014Z "digest": "sha256:43dbc789bd58614201475dab5219ac163330be202d034a5464531e9d1e446a6a" 2024-06-01T03:31:07.4521623Z }, 2024-06-01T03:31:07.4521866Z { 2024-06-01T03:31:07.4522422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4522982Z "size": 1443748612, 2024-06-01T03:31:07.4523544Z "digest": "sha256:76c0a5120664ef0f89af07fcaf00baf85222f83e454488eb009309386d773e64" 2024-06-01T03:31:07.4524154Z }, 2024-06-01T03:31:07.4524392Z { 2024-06-01T03:31:07.4524819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4525367Z "size": 106, 2024-06-01T03:31:07.4525911Z "digest": "sha256:98d8a99ae4e01f8c2c6a24c2b4ae8060aaeec8b4e06c94975b8ba03fadf1a45d" 2024-06-01T03:31:07.4526535Z }, 2024-06-01T03:31:07.4526777Z { 2024-06-01T03:31:07.4527204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4527767Z "size": 558, 2024-06-01T03:31:07.4528341Z "digest": "sha256:1633e81ae8df2c80585453a900c345d81f2aef6832ea1f321ef4cfdb46a79e29" 2024-06-01T03:31:07.4528954Z }, 2024-06-01T03:31:07.4529199Z { 2024-06-01T03:31:07.4529628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4530334Z "size": 46248265, 2024-06-01T03:31:07.4530888Z "digest": "sha256:5b69709a20d2572f04ef231bb97689f7e16a7f8217e2dc68f3951aac42dfb6c4" 2024-06-01T03:31:07.4531506Z }, 2024-06-01T03:31:07.4531746Z { 2024-06-01T03:31:07.4532179Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-01T03:31:07.4532736Z "size": 111, 2024-06-01T03:31:07.4533286Z "digest": "sha256:85ee5de1de27d907dede0b393f7d70d961096eb641707fbceed5cbdbe2bd3d85" 2024-06-01T03:31:07.4533926Z } 2024-06-01T03:31:07.4534170Z ] 2024-06-01T03:31:07.4534418Z } 2024-06-01T03:31:07.4634562Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-01T03:31:07.4635062Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-01T03:31:07.4635616Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-01T03:31:07.4643478Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:07.4643993Z env: 2024-06-01T03:31:07.4644278Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:07.4645498Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.4646733Z ##[endgroup] 2024-06-01T03:31:07.4668912Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks-7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.4727995Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-01T03:31:07.4728636Z with: 2024-06-01T03:31:07.4729748Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.4731120Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.4731645Z env: 2024-06-01T03:31:07.4731921Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:07.4732269Z ##[endgroup] 2024-06-01T03:31:07.4748270Z ##[group]Run set -x 2024-06-01T03:31:07.4748631Z set -x 2024-06-01T03:31:07.4748936Z set +e 2024-06-01T03:31:07.4749236Z  2024-06-01T03:31:07.4749511Z login() { 2024-06-01T03:31:07.4750363Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-01T03:31:07.4751098Z } 2024-06-01T03:31:07.4751384Z  2024-06-01T03:31:07.4751691Z retry () { 2024-06-01T03:31:07.4752086Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-01T03:31:07.4752535Z } 2024-06-01T03:31:07.4752813Z  2024-06-01T03:31:07.4753135Z retry login "${DOCKER_REGISTRY}" 2024-06-01T03:31:07.4753550Z  2024-06-01T03:31:07.4753830Z set -e 2024-06-01T03:31:07.4754310Z # ignore output since only exit code is used for conditional 2024-06-01T03:31:07.4755028Z # only pull docker image if it's not available locally 2024-06-01T03:31:07.4755804Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-01T03:31:07.4756522Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-01T03:31:07.4756967Z fi 2024-06-01T03:31:07.4764318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:31:07.4764810Z env: 2024-06-01T03:31:07.4765086Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:31:07.4766249Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.4767572Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.4768121Z ##[endgroup] 2024-06-01T03:31:07.4787394Z + set +e 2024-06-01T03:31:07.4788031Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.4788716Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.4789457Z + aws ecr get-login-password --region us-east-1 2024-06-01T03:31:07.4790812Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-01T03:31:07.8943148Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-01T03:31:07.8944180Z Configure a credential helper to remove this warning. See 2024-06-01T03:31:07.8945083Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-01T03:31:07.8945634Z 2024-06-01T03:31:07.8945757Z Login Succeeded 2024-06-01T03:31:07.8955110Z + set -e 2024-06-01T03:31:07.8956652Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.9110359Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:07.9112566Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:31:08.1637976Z 7790448f81f0f3396d69a76eba86a4be7ac35343: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks 2024-06-01T03:31:08.1642008Z 63e9bbe32327: Pulling fs layer 2024-06-01T03:31:08.1644606Z cfb3d849840e: Pulling fs layer 2024-06-01T03:31:08.1645501Z 968831e596a6: Pulling fs layer 2024-06-01T03:31:08.1646218Z ea310eb267ca: Pulling fs layer 2024-06-01T03:31:08.1646817Z 3af11d09e9cd: Pulling fs layer 2024-06-01T03:31:08.1647385Z ebfec18059b9: Pulling fs layer 2024-06-01T03:31:08.1647936Z 533b4aebf169: Pulling fs layer 2024-06-01T03:31:08.1648497Z 9dd75d06a091: Pulling fs layer 2024-06-01T03:31:08.1649055Z 30bfca4dd349: Pulling fs layer 2024-06-01T03:31:08.1649597Z 1b57ce94cad9: Pulling fs layer 2024-06-01T03:31:08.1650135Z 9ee6bdb31195: Pulling fs layer 2024-06-01T03:31:08.1650524Z fec42e0e4ca2: Pulling fs layer 2024-06-01T03:31:08.1650914Z d0d1bc0363fb: Pulling fs layer 2024-06-01T03:31:08.1651291Z ea310eb267ca: Waiting 2024-06-01T03:31:08.1651634Z cc64afd638a7: Pulling fs layer 2024-06-01T03:31:08.1652015Z 52f0d9fdf396: Pulling fs layer 2024-06-01T03:31:08.1652386Z 65a297b68fd1: Pulling fs layer 2024-06-01T03:31:08.1652751Z 3af11d09e9cd: Waiting 2024-06-01T03:31:08.1653087Z 424e127052d0: Pulling fs layer 2024-06-01T03:31:08.1653452Z 9dd75d06a091: Waiting 2024-06-01T03:31:08.1653797Z 184e6daf3e53: Pulling fs layer 2024-06-01T03:31:08.1654229Z 9ee6bdb31195: Waiting 2024-06-01T03:31:08.1654564Z 7a163619b003: Pulling fs layer 2024-06-01T03:31:08.1654996Z fec42e0e4ca2: Waiting 2024-06-01T03:31:08.1655336Z 2f1d5eae98e5: Pulling fs layer 2024-06-01T03:31:08.1655702Z ebfec18059b9: Waiting 2024-06-01T03:31:08.1656042Z 22ed7fbe39a2: Pulling fs layer 2024-06-01T03:31:08.1656395Z 52f0d9fdf396: Waiting 2024-06-01T03:31:08.1656725Z 89b2bc3b8262: Pulling fs layer 2024-06-01T03:31:08.1657086Z d0d1bc0363fb: Waiting 2024-06-01T03:31:08.1657420Z fc61ad516d94: Pulling fs layer 2024-06-01T03:31:08.1657793Z 65a297b68fd1: Waiting 2024-06-01T03:31:08.1658117Z 22ed7fbe39a2: Waiting 2024-06-01T03:31:08.1658437Z 1b57ce94cad9: Waiting 2024-06-01T03:31:08.1658742Z 7a163619b003: Waiting 2024-06-01T03:31:08.1659053Z 184e6daf3e53: Waiting 2024-06-01T03:31:08.1659390Z 1aafe81a9a9a: Pulling fs layer 2024-06-01T03:31:08.1659754Z 424e127052d0: Waiting 2024-06-01T03:31:08.1660079Z 9087b5d4aba0: Pulling fs layer 2024-06-01T03:31:08.1660454Z 41894f89ccf0: Pulling fs layer 2024-06-01T03:31:08.1660867Z 82ae7e4ef3ff: Pulling fs layer 2024-06-01T03:31:08.1661234Z 2f1d5eae98e5: Waiting 2024-06-01T03:31:08.1661552Z cc64afd638a7: Waiting 2024-06-01T03:31:08.1661878Z 080b4f321816: Pulling fs layer 2024-06-01T03:31:08.1662252Z 9f98bff3bbd5: Pulling fs layer 2024-06-01T03:31:08.1662613Z 41894f89ccf0: Waiting 2024-06-01T03:31:08.1663004Z c8515406c82c: Pulling fs layer 2024-06-01T03:31:08.1663386Z dba13cb5868e: Pulling fs layer 2024-06-01T03:31:08.1663819Z 82ae7e4ef3ff: Waiting 2024-06-01T03:31:08.1664312Z 9f98bff3bbd5: Waiting 2024-06-01T03:31:08.1664704Z 9087b5d4aba0: Waiting 2024-06-01T03:31:08.1665038Z f2036f97fbab: Pulling fs layer 2024-06-01T03:31:08.1665398Z c8515406c82c: Waiting 2024-06-01T03:31:08.1665781Z 1aafe81a9a9a: Waiting 2024-06-01T03:31:08.1666108Z dba13cb5868e: Waiting 2024-06-01T03:31:08.1666457Z de273ac17579: Pulling fs layer 2024-06-01T03:31:08.1666890Z 29ca9539f3ae: Pulling fs layer 2024-06-01T03:31:08.1667267Z 9f31f86a8b95: Pulling fs layer 2024-06-01T03:31:08.1667723Z d2455d25abe3: Pulling fs layer 2024-06-01T03:31:08.1668111Z 859146fb20cc: Pulling fs layer 2024-06-01T03:31:08.1668478Z 29ca9539f3ae: Waiting 2024-06-01T03:31:08.1668874Z d401e21441f6: Pulling fs layer 2024-06-01T03:31:08.1669238Z 9f31f86a8b95: Waiting 2024-06-01T03:31:08.1669644Z f80c9a959f81: Pulling fs layer 2024-06-01T03:31:08.1670194Z f2036f97fbab: Waiting 2024-06-01T03:31:08.1670597Z 5ad81974f126: Pulling fs layer 2024-06-01T03:31:08.1670958Z f80c9a959f81: Waiting 2024-06-01T03:31:08.1671306Z 859146fb20cc: Waiting 2024-06-01T03:31:08.1671680Z 25829e6ccd20: Pulling fs layer 2024-06-01T03:31:08.1672089Z 672438d05e4a: Pulling fs layer 2024-06-01T03:31:08.1672494Z e5eecd30f362: Pulling fs layer 2024-06-01T03:31:08.1672903Z 23b679499cb9: Pulling fs layer 2024-06-01T03:31:08.1673294Z 25829e6ccd20: Waiting 2024-06-01T03:31:08.1673608Z 672438d05e4a: Waiting 2024-06-01T03:31:08.1674016Z e139f208fe91: Pulling fs layer 2024-06-01T03:31:08.1674510Z 4c505cd63777: Pulling fs layer 2024-06-01T03:31:08.1674966Z d432f274bf71: Pulling fs layer 2024-06-01T03:31:08.1675361Z 23b679499cb9: Waiting 2024-06-01T03:31:08.1675762Z a960cc0a9498: Pulling fs layer 2024-06-01T03:31:08.1676140Z 333ad8bb3f6e: Pulling fs layer 2024-06-01T03:31:08.1676587Z d5d8665989ed: Pulling fs layer 2024-06-01T03:31:08.1676962Z a5d8e3034f56: Pulling fs layer 2024-06-01T03:31:08.1677382Z e139f208fe91: Waiting 2024-06-01T03:31:08.1677721Z 292a840d6192: Pulling fs layer 2024-06-01T03:31:08.1678124Z 4c505cd63777: Waiting 2024-06-01T03:31:08.1678530Z 545ad07eb1b0: Pulling fs layer 2024-06-01T03:31:08.1678913Z 69b414c0796e: Pulling fs layer 2024-06-01T03:31:08.1679352Z 20566ec9e819: Pulling fs layer 2024-06-01T03:31:08.1679715Z d432f274bf71: Waiting 2024-06-01T03:31:08.1680117Z dfba59f41e5c: Pulling fs layer 2024-06-01T03:31:08.1680480Z a960cc0a9498: Waiting 2024-06-01T03:31:08.1680879Z ad96f96c6a61: Pulling fs layer 2024-06-01T03:31:08.1681271Z 3064c9882e6e: Pulling fs layer 2024-06-01T03:31:08.1681708Z eb8bda3e2c43: Pulling fs layer 2024-06-01T03:31:08.1682076Z a5d8e3034f56: Waiting 2024-06-01T03:31:08.1682572Z 545ad07eb1b0: Waiting 2024-06-01T03:31:08.1682908Z 292a840d6192: Waiting 2024-06-01T03:31:08.1683230Z 05ed995317da: Pulling fs layer 2024-06-01T03:31:08.1683656Z 69b414c0796e: Waiting 2024-06-01T03:31:08.1683985Z 178627ca8d56: Pulling fs layer 2024-06-01T03:31:08.1684424Z 43dbc789bd58: Pulling fs layer 2024-06-01T03:31:08.1684800Z 76c0a5120664: Pulling fs layer 2024-06-01T03:31:08.1685242Z 98d8a99ae4e0: Pulling fs layer 2024-06-01T03:31:08.1685605Z d5d8665989ed: Waiting 2024-06-01T03:31:08.1685983Z 1633e81ae8df: Pulling fs layer 2024-06-01T03:31:08.1686367Z 20566ec9e819: Waiting 2024-06-01T03:31:08.1686686Z 3064c9882e6e: Waiting 2024-06-01T03:31:08.1687064Z 05ed995317da: Waiting 2024-06-01T03:31:08.1687371Z 178627ca8d56: Waiting 2024-06-01T03:31:08.1687745Z ad96f96c6a61: Waiting 2024-06-01T03:31:08.1688096Z 5b69709a20d2: Pulling fs layer 2024-06-01T03:31:08.1688491Z eb8bda3e2c43: Waiting 2024-06-01T03:31:08.1688846Z 43dbc789bd58: Waiting 2024-06-01T03:31:08.1689164Z 1633e81ae8df: Waiting 2024-06-01T03:31:08.1689548Z 98d8a99ae4e0: Waiting 2024-06-01T03:31:08.1689886Z 76c0a5120664: Waiting 2024-06-01T03:31:08.1690254Z 85ee5de1de27: Pulling fs layer 2024-06-01T03:31:08.1690652Z 5b69709a20d2: Waiting 2024-06-01T03:31:08.3053937Z cfb3d849840e: Verifying Checksum 2024-06-01T03:31:08.3054464Z cfb3d849840e: Download complete 2024-06-01T03:31:08.3722717Z ea310eb267ca: Verifying Checksum 2024-06-01T03:31:08.3723170Z ea310eb267ca: Download complete 2024-06-01T03:31:08.4513136Z 3af11d09e9cd: Verifying Checksum 2024-06-01T03:31:08.4513912Z 3af11d09e9cd: Download complete 2024-06-01T03:31:08.5097194Z 63e9bbe32327: Verifying Checksum 2024-06-01T03:31:08.5097664Z 63e9bbe32327: Download complete 2024-06-01T03:31:08.5846837Z 533b4aebf169: Verifying Checksum 2024-06-01T03:31:08.5847308Z 533b4aebf169: Download complete 2024-06-01T03:31:08.6572402Z 9dd75d06a091: Verifying Checksum 2024-06-01T03:31:08.6572855Z 9dd75d06a091: Download complete 2024-06-01T03:31:08.7317960Z 30bfca4dd349: Download complete 2024-06-01T03:31:08.7960561Z 968831e596a6: Verifying Checksum 2024-06-01T03:31:08.7961018Z 968831e596a6: Download complete 2024-06-01T03:31:08.8702368Z 9ee6bdb31195: Download complete 2024-06-01T03:31:08.9369271Z fec42e0e4ca2: Download complete 2024-06-01T03:31:09.1740881Z 63e9bbe32327: Pull complete 2024-06-01T03:31:09.4022344Z cfb3d849840e: Pull complete 2024-06-01T03:31:10.2772663Z 968831e596a6: Pull complete 2024-06-01T03:31:10.3674539Z ea310eb267ca: Pull complete 2024-06-01T03:31:10.4719918Z 3af11d09e9cd: Pull complete 2024-06-01T03:31:11.4530178Z d0d1bc0363fb: Verifying Checksum 2024-06-01T03:31:11.4530861Z d0d1bc0363fb: Download complete 2024-06-01T03:31:11.5303171Z cc64afd638a7: Verifying Checksum 2024-06-01T03:31:11.5303681Z cc64afd638a7: Download complete 2024-06-01T03:31:11.6065788Z 52f0d9fdf396: Download complete 2024-06-01T03:31:11.6784096Z 65a297b68fd1: Verifying Checksum 2024-06-01T03:31:11.6784751Z 65a297b68fd1: Download complete 2024-06-01T03:31:12.6407711Z 424e127052d0: Verifying Checksum 2024-06-01T03:31:12.6408199Z 424e127052d0: Download complete 2024-06-01T03:31:12.7358402Z 184e6daf3e53: Verifying Checksum 2024-06-01T03:31:12.7358939Z 184e6daf3e53: Download complete 2024-06-01T03:31:12.8117560Z 7a163619b003: Verifying Checksum 2024-06-01T03:31:12.8118040Z 7a163619b003: Download complete 2024-06-01T03:31:12.8833995Z 2f1d5eae98e5: Verifying Checksum 2024-06-01T03:31:12.8834586Z 2f1d5eae98e5: Download complete 2024-06-01T03:31:22.1173032Z ebfec18059b9: Verifying Checksum 2024-06-01T03:31:22.1173546Z ebfec18059b9: Download complete 2024-06-01T03:31:22.1965631Z 89b2bc3b8262: Verifying Checksum 2024-06-01T03:31:22.1966274Z 89b2bc3b8262: Download complete 2024-06-01T03:31:22.2733351Z fc61ad516d94: Download complete 2024-06-01T03:31:22.3908972Z 1aafe81a9a9a: Download complete 2024-06-01T03:31:22.4764678Z 9087b5d4aba0: Verifying Checksum 2024-06-01T03:31:22.4765370Z 9087b5d4aba0: Download complete 2024-06-01T03:31:22.5552188Z 41894f89ccf0: Download complete 2024-06-01T03:31:23.8240945Z 82ae7e4ef3ff: Verifying Checksum 2024-06-01T03:31:23.8241433Z 82ae7e4ef3ff: Download complete 2024-06-01T03:31:23.9060978Z 080b4f321816: Verifying Checksum 2024-06-01T03:31:23.9869243Z 080b4f321816: Download complete 2024-06-01T03:31:23.9869686Z 9f98bff3bbd5: Download complete 2024-06-01T03:31:24.0434142Z c8515406c82c: Verifying Checksum 2024-06-01T03:31:24.0434568Z c8515406c82c: Download complete 2024-06-01T03:31:24.1250426Z dba13cb5868e: Verifying Checksum 2024-06-01T03:31:24.1250949Z dba13cb5868e: Download complete 2024-06-01T03:31:24.2048621Z f2036f97fbab: Verifying Checksum 2024-06-01T03:31:24.2049472Z f2036f97fbab: Download complete 2024-06-01T03:31:28.7261362Z de273ac17579: Verifying Checksum 2024-06-01T03:31:28.7261961Z de273ac17579: Download complete 2024-06-01T03:31:28.8004677Z 29ca9539f3ae: Verifying Checksum 2024-06-01T03:31:28.8005129Z 29ca9539f3ae: Download complete 2024-06-01T03:31:29.8954809Z 9f31f86a8b95: Verifying Checksum 2024-06-01T03:31:29.8955369Z 9f31f86a8b95: Download complete 2024-06-01T03:31:30.3024668Z d2455d25abe3: Verifying Checksum 2024-06-01T03:31:30.3025271Z d2455d25abe3: Download complete 2024-06-01T03:31:30.3875063Z 859146fb20cc: Download complete 2024-06-01T03:31:30.4433680Z d401e21441f6: Verifying Checksum 2024-06-01T03:31:30.4434242Z d401e21441f6: Download complete 2024-06-01T03:31:30.7064497Z f80c9a959f81: Verifying Checksum 2024-06-01T03:31:30.7064951Z f80c9a959f81: Download complete 2024-06-01T03:31:30.7841182Z 5ad81974f126: Download complete 2024-06-01T03:31:30.8696251Z 25829e6ccd20: Verifying Checksum 2024-06-01T03:31:30.8696877Z 25829e6ccd20: Download complete 2024-06-01T03:31:30.9518880Z 672438d05e4a: Download complete 2024-06-01T03:31:34.2430947Z ebfec18059b9: Pull complete 2024-06-01T03:31:34.4752655Z 533b4aebf169: Pull complete 2024-06-01T03:31:34.6617976Z 9dd75d06a091: Pull complete 2024-06-01T03:31:34.8679308Z 1b57ce94cad9: Verifying Checksum 2024-06-01T03:31:34.8679796Z 1b57ce94cad9: Download complete 2024-06-01T03:31:34.9055365Z 30bfca4dd349: Pull complete 2024-06-01T03:31:34.9428552Z 23b679499cb9: Verifying Checksum 2024-06-01T03:31:34.9428994Z 23b679499cb9: Download complete 2024-06-01T03:31:35.0134819Z e139f208fe91: Verifying Checksum 2024-06-01T03:31:35.0135293Z e139f208fe91: Download complete 2024-06-01T03:31:35.4983943Z 4c505cd63777: Verifying Checksum 2024-06-01T03:31:35.4984540Z 4c505cd63777: Download complete 2024-06-01T03:31:35.5693944Z d432f274bf71: Verifying Checksum 2024-06-01T03:31:35.5694416Z d432f274bf71: Download complete 2024-06-01T03:31:35.6382895Z a960cc0a9498: Verifying Checksum 2024-06-01T03:31:35.6383339Z a960cc0a9498: Download complete 2024-06-01T03:31:35.7110271Z 333ad8bb3f6e: Download complete 2024-06-01T03:31:35.7804037Z d5d8665989ed: Verifying Checksum 2024-06-01T03:31:35.7804477Z d5d8665989ed: Download complete 2024-06-01T03:31:40.9125540Z 22ed7fbe39a2: Verifying Checksum 2024-06-01T03:31:40.9126209Z 22ed7fbe39a2: Download complete 2024-06-01T03:31:40.9830402Z 292a840d6192: Verifying Checksum 2024-06-01T03:31:40.9831026Z 292a840d6192: Download complete 2024-06-01T03:31:41.0578963Z 545ad07eb1b0: Verifying Checksum 2024-06-01T03:31:41.0579554Z 545ad07eb1b0: Download complete 2024-06-01T03:31:41.1816458Z 69b414c0796e: Verifying Checksum 2024-06-01T03:31:41.1817108Z 69b414c0796e: Download complete 2024-06-01T03:31:41.2448831Z 20566ec9e819: Verifying Checksum 2024-06-01T03:31:41.2449382Z 20566ec9e819: Download complete 2024-06-01T03:31:41.3146426Z dfba59f41e5c: Verifying Checksum 2024-06-01T03:31:41.3146976Z dfba59f41e5c: Download complete 2024-06-01T03:31:41.4144121Z ad96f96c6a61: Download complete 2024-06-01T03:31:41.4948923Z 3064c9882e6e: Verifying Checksum 2024-06-01T03:31:41.4949514Z 3064c9882e6e: Download complete 2024-06-01T03:31:41.6223453Z eb8bda3e2c43: Verifying Checksum 2024-06-01T03:31:41.6224031Z eb8bda3e2c43: Download complete 2024-06-01T03:31:41.6908969Z 05ed995317da: Verifying Checksum 2024-06-01T03:31:41.6909493Z 05ed995317da: Download complete 2024-06-01T03:31:42.2875959Z 178627ca8d56: Verifying Checksum 2024-06-01T03:31:42.2876905Z 178627ca8d56: Download complete 2024-06-01T03:31:42.3618295Z 43dbc789bd58: Verifying Checksum 2024-06-01T03:31:42.3618830Z 43dbc789bd58: Download complete 2024-06-01T03:32:07.7246797Z a5d8e3034f56: Download complete 2024-06-01T03:32:07.8152088Z 98d8a99ae4e0: Download complete 2024-06-01T03:32:07.8951519Z 1633e81ae8df: Verifying Checksum 2024-06-01T03:32:07.8953358Z 1633e81ae8df: Download complete 2024-06-01T03:32:09.1240414Z 5b69709a20d2: Verifying Checksum 2024-06-01T03:32:09.1240886Z 5b69709a20d2: Download complete 2024-06-01T03:32:09.2178105Z 85ee5de1de27: Verifying Checksum 2024-06-01T03:32:09.2178541Z 85ee5de1de27: Download complete 2024-06-01T03:32:12.5822259Z 76c0a5120664: Verifying Checksum 2024-06-01T03:32:12.5822726Z 76c0a5120664: Download complete 2024-06-01T03:32:37.3534503Z e5eecd30f362: Verifying Checksum 2024-06-01T03:32:37.3534950Z e5eecd30f362: Download complete 2024-06-01T03:32:53.0243696Z 1b57ce94cad9: Pull complete 2024-06-01T03:32:53.2441003Z 9ee6bdb31195: Pull complete 2024-06-01T03:32:53.4782541Z fec42e0e4ca2: Pull complete 2024-06-01T03:32:58.5105181Z d0d1bc0363fb: Pull complete 2024-06-01T03:32:58.7438576Z cc64afd638a7: Pull complete 2024-06-01T03:32:58.9781314Z 52f0d9fdf396: Pull complete 2024-06-01T03:32:59.2020719Z 65a297b68fd1: Pull complete 2024-06-01T03:33:01.1021732Z 424e127052d0: Pull complete 2024-06-01T03:33:01.3319321Z 184e6daf3e53: Pull complete 2024-06-01T03:33:01.5663376Z 7a163619b003: Pull complete 2024-06-01T03:33:01.7772542Z 2f1d5eae98e5: Pull complete 2024-06-01T03:33:38.9530452Z 22ed7fbe39a2: Pull complete 2024-06-01T03:33:39.1667912Z 89b2bc3b8262: Pull complete 2024-06-01T03:33:39.3882487Z fc61ad516d94: Pull complete 2024-06-01T03:33:39.5549755Z 1aafe81a9a9a: Pull complete 2024-06-01T03:33:39.7042536Z 9087b5d4aba0: Pull complete 2024-06-01T03:33:39.9221500Z 41894f89ccf0: Pull complete 2024-06-01T03:33:42.9960851Z 82ae7e4ef3ff: Pull complete 2024-06-01T03:33:43.1302992Z 080b4f321816: Pull complete 2024-06-01T03:33:43.2918649Z 9f98bff3bbd5: Pull complete 2024-06-01T03:33:43.4720398Z c8515406c82c: Pull complete 2024-06-01T03:33:43.6934963Z dba13cb5868e: Pull complete 2024-06-01T03:33:43.8729405Z f2036f97fbab: Pull complete 2024-06-01T03:33:49.8848345Z de273ac17579: Pull complete 2024-06-01T03:33:50.1031953Z 29ca9539f3ae: Pull complete 2024-06-01T03:33:50.3275657Z 9f31f86a8b95: Pull complete 2024-06-01T03:33:51.2077272Z d2455d25abe3: Pull complete 2024-06-01T03:33:51.3903947Z 859146fb20cc: Pull complete 2024-06-01T03:33:51.6376077Z d401e21441f6: Pull complete 2024-06-01T03:33:52.0790748Z f80c9a959f81: Pull complete 2024-06-01T03:33:52.2950427Z 5ad81974f126: Pull complete 2024-06-01T03:33:52.7486358Z 25829e6ccd20: Pull complete 2024-06-01T03:33:52.9708359Z 672438d05e4a: Pull complete 2024-06-01T03:34:29.4515405Z e5eecd30f362: Pull complete 2024-06-01T03:34:29.6530930Z 23b679499cb9: Pull complete 2024-06-01T03:34:29.8368042Z e139f208fe91: Pull complete 2024-06-01T03:34:31.1294332Z 4c505cd63777: Pull complete 2024-06-01T03:34:31.2786548Z d432f274bf71: Pull complete 2024-06-01T03:34:31.4375369Z a960cc0a9498: Pull complete 2024-06-01T03:34:31.8110438Z 333ad8bb3f6e: Pull complete 2024-06-01T03:34:31.9378202Z d5d8665989ed: Pull complete 2024-06-01T03:35:01.7898256Z a5d8e3034f56: Pull complete 2024-06-01T03:35:02.0359626Z 292a840d6192: Pull complete 2024-06-01T03:35:02.2207643Z 545ad07eb1b0: Pull complete 2024-06-01T03:35:02.5494874Z 69b414c0796e: Pull complete 2024-06-01T03:35:02.7756298Z 20566ec9e819: Pull complete 2024-06-01T03:35:03.0003175Z dfba59f41e5c: Pull complete 2024-06-01T03:35:03.2334437Z ad96f96c6a61: Pull complete 2024-06-01T03:35:03.4300601Z 3064c9882e6e: Pull complete 2024-06-01T03:35:04.0255964Z eb8bda3e2c43: Pull complete 2024-06-01T03:35:04.2608059Z 05ed995317da: Pull complete 2024-06-01T03:35:05.9452654Z 178627ca8d56: Pull complete 2024-06-01T03:35:06.1583236Z 43dbc789bd58: Pull complete 2024-06-01T03:35:20.3892423Z 76c0a5120664: Pull complete 2024-06-01T03:35:20.6111025Z 98d8a99ae4e0: Pull complete 2024-06-01T03:35:20.8308460Z 1633e81ae8df: Pull complete 2024-06-01T03:35:21.6680899Z 5b69709a20d2: Pull complete 2024-06-01T03:35:21.8922161Z 85ee5de1de27: Pull complete 2024-06-01T03:35:22.0025354Z Digest: sha256:b8296027e5f1cfef6dbf6c011e0b1f6ebfa606876a8eb59b07c03b8046ec8490 2024-06-01T03:35:22.0472638Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:35:22.0690958Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:35:22.0734341Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-01T03:35:22.0735307Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-01T03:35:22.0743320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:35:22.0743835Z env: 2024-06-01T03:35:22.0744104Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:35:22.0744457Z ##[endgroup] 2024-06-01T03:35:22.0901657Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-01T03:35:22.0902224Z with: 2024-06-01T03:35:22.0902515Z driver-version: 550.54.15 2024-06-01T03:35:22.0902865Z env: 2024-06-01T03:35:22.0903134Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:35:22.0903487Z ##[endgroup] 2024-06-01T03:35:22.1044327Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-01T03:35:22.1045060Z with: 2024-06-01T03:35:22.1045342Z timeout_minutes: 10 2024-06-01T03:35:22.1045675Z max_attempts: 3 2024-06-01T03:35:22.1077392Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-01T03:35:22.1109711Z retry_wait_seconds: 10 2024-06-01T03:35:22.1110295Z polling_interval_seconds: 1 2024-06-01T03:35:22.1110710Z warning_on_retry: true 2024-06-01T03:35:22.1111069Z continue_on_error: false 2024-06-01T03:35:22.1111409Z env: 2024-06-01T03:35:22.1111682Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:35:22.1112037Z DRIVER_VERSION: 550.54.15 2024-06-01T03:35:22.1112393Z ##[endgroup] 2024-06-01T03:35:22.1637538Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-01T03:35:22.1638501Z + pre_install_nvidia_driver_amzn2 2024-06-01T03:35:22.1639237Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-01T03:35:22.4429656Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-01T03:35:22.4782131Z No Match for argument: nvidia-driver-latest-dkms 2024-06-01T03:35:22.4990255Z No Packages marked for removal 2024-06-01T03:35:22.5107843Z + install_nvidia_driver_common 2024-06-01T03:35:22.5110482Z + echo 'Before installing NVIDIA driver' 2024-06-01T03:35:22.5110986Z Before installing NVIDIA driver 2024-06-01T03:35:22.5112529Z + lspci 2024-06-01T03:35:22.5958490Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-01T03:35:22.5959297Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-01T03:35:22.5960254Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-01T03:35:22.5961094Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-01T03:35:22.5961890Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-01T03:35:22.5962770Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-01T03:35:22.5963634Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-01T03:35:22.5964452Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-01T03:35:22.5965065Z + lsmod 2024-06-01T03:35:22.5971048Z Module Size Used by 2024-06-01T03:35:22.5971541Z backlight 16384 0 2024-06-01T03:35:22.5972059Z xt_conntrack 16384 1 2024-06-01T03:35:22.5972501Z ipt_MASQUERADE 16384 1 2024-06-01T03:35:22.5972961Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-01T03:35:22.5973597Z nf_conntrack_netlink 49152 0 2024-06-01T03:35:22.5974207Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-01T03:35:22.5974756Z xfrm_user 45056 1 2024-06-01T03:35:22.5975165Z xfrm_algo 16384 1 xfrm_user 2024-06-01T03:35:22.5975789Z iptable_nat 16384 1 2024-06-01T03:35:22.5976186Z nf_conntrack_ipv4 16384 3 2024-06-01T03:35:22.5976638Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-01T03:35:22.5977283Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-01T03:35:22.5977957Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-01T03:35:22.5978934Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-01T03:35:22.5980035Z xt_addrtype 16384 2 2024-06-01T03:35:22.5980595Z iptable_filter 16384 1 2024-06-01T03:35:22.5981038Z br_netfilter 24576 0 2024-06-01T03:35:22.5981449Z bridge 172032 1 br_netfilter 2024-06-01T03:35:22.5981903Z stp 16384 1 bridge 2024-06-01T03:35:22.5982332Z llc 16384 2 bridge,stp 2024-06-01T03:35:22.5982764Z overlay 86016 0 2024-06-01T03:35:22.5983225Z sunrpc 393216 1 2024-06-01T03:35:22.5983611Z dm_mirror 28672 0 2024-06-01T03:35:22.6024409Z dm_region_hash 20480 1 dm_mirror 2024-06-01T03:35:22.6025443Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-01T03:35:22.6026065Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-01T03:35:22.6026525Z dax 69632 1 dm_mod 2024-06-01T03:35:22.6026942Z crc32_pclmul 16384 0 2024-06-01T03:35:22.6027347Z ghash_clmulni_intel 16384 0 2024-06-01T03:35:22.6027734Z pcbc 16384 0 2024-06-01T03:35:22.6028111Z aesni_intel 188416 0 2024-06-01T03:35:22.6028509Z aes_x86_64 20480 1 aesni_intel 2024-06-01T03:35:22.6028959Z crypto_simd 16384 1 aesni_intel 2024-06-01T03:35:22.6029416Z glue_helper 16384 1 aesni_intel 2024-06-01T03:35:22.6030209Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-01T03:35:22.6030777Z mousedev 24576 0 2024-06-01T03:35:22.6031161Z evdev 20480 3 2024-06-01T03:35:22.6031531Z psmouse 32768 0 2024-06-01T03:35:22.6031903Z button 16384 0 2024-06-01T03:35:22.6032262Z ena 139264 0 2024-06-01T03:35:22.6032640Z ptp 20480 1 ena 2024-06-01T03:35:22.6033042Z pps_core 20480 1 ptp 2024-06-01T03:35:22.6033445Z crc32c_intel 24576 0 2024-06-01T03:35:22.6033811Z autofs4 49152 2 2024-06-01T03:35:22.6034185Z + modinfo nvidia 2024-06-01T03:35:22.6034934Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-01T03:35:22.6035641Z alias: char-major-195-* 2024-06-01T03:35:22.6036020Z version: 550.54.15 2024-06-01T03:35:22.6036372Z supported: external 2024-06-01T03:35:22.6036714Z license: NVIDIA 2024-06-01T03:35:22.6037087Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-01T03:35:22.6037584Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-01T03:35:22.6038057Z srcversion: 833721318DA517F0C2FEC97 2024-06-01T03:35:22.6038531Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-01T03:35:22.6039042Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-01T03:35:22.6039551Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-01T03:35:22.6040041Z depends: i2c-core,drm 2024-06-01T03:35:22.6040407Z retpoline: Y 2024-06-01T03:35:22.6040708Z name: nvidia 2024-06-01T03:35:22.6041326Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-01T03:35:22.6042001Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-01T03:35:22.6042667Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-01T03:35:22.6043417Z parm: NVreg_ResmanDebugLevel:int 2024-06-01T03:35:22.6043873Z parm: NVreg_RmLogonRC:int 2024-06-01T03:35:22.6044336Z parm: NVreg_ModifyDeviceFiles:int 2024-06-01T03:35:22.6075907Z parm: NVreg_DeviceFileUID:int 2024-06-01T03:35:22.6076720Z parm: NVreg_DeviceFileGID:int 2024-06-01T03:35:22.6077314Z parm: NVreg_DeviceFileMode:int 2024-06-01T03:35:22.6077864Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-01T03:35:22.6078446Z parm: NVreg_UsePageAttributeTable:int 2024-06-01T03:35:22.6078938Z parm: NVreg_EnablePCIeGen3:int 2024-06-01T03:35:22.6079378Z parm: NVreg_EnableMSI:int 2024-06-01T03:35:22.6079895Z parm: NVreg_TCEBypassMode:int 2024-06-01T03:35:22.6080371Z parm: NVreg_EnableStreamMemOPs:int 2024-06-01T03:35:22.6080940Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-01T03:35:22.6081562Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-01T03:35:22.6082126Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-01T03:35:22.6082749Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-01T03:35:22.6083529Z parm: NVreg_DynamicPowerManagement:int 2024-06-01T03:35:22.6084164Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-01T03:35:22.6084765Z parm: NVreg_EnableGpuFirmware:int 2024-06-01T03:35:22.6085269Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-01T03:35:22.6085813Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-01T03:35:22.6086369Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-01T03:35:22.6086869Z parm: NVreg_MemoryPoolSize:int 2024-06-01T03:35:22.6087351Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-01T03:35:22.6087844Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-01T03:35:22.6088332Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-01T03:35:22.6088950Z parm: NVreg_NvLinkDisable:int 2024-06-01T03:35:22.6089473Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-01T03:35:22.6090008Z parm: NVreg_RegisterPCIDriver:int 2024-06-01T03:35:22.6090534Z parm: NVreg_EnableResizableBar:int 2024-06-01T03:35:22.6091047Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-01T03:35:22.6091560Z parm: NVreg_EnableNonblockingOpen:int 2024-06-01T03:35:22.6092052Z parm: NVreg_RegistryDwords:charp 2024-06-01T03:35:22.6092551Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-01T03:35:22.6093039Z parm: NVreg_RmMsg:charp 2024-06-01T03:35:22.6093457Z parm: NVreg_GpuBlacklist:charp 2024-06-01T03:35:22.6093931Z parm: NVreg_TemporaryFilePath:charp 2024-06-01T03:35:22.6094416Z parm: NVreg_ExcludedGpus:charp 2024-06-01T03:35:22.6094883Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-01T03:35:22.6095371Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-01T03:35:22.6095854Z parm: NVreg_ImexChannelCount:int 2024-06-01T03:35:22.6096308Z parm: rm_firmware_active:charp 2024-06-01T03:35:22.6096731Z + HAS_NVIDIA_DRIVER=0 2024-06-01T03:35:22.6097134Z ++ command -v nvidia-smi 2024-06-01T03:35:22.6097523Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-01T03:35:22.6097908Z + set +e 2024-06-01T03:35:22.6098418Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-01T03:35:24.6367116Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-01T03:35:24.6367801Z + NVIDIA_SMI_STATUS=0 2024-06-01T03:35:24.6368444Z + '[' 0 -ne 0 ']' 2024-06-01T03:35:24.6368822Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-01T03:35:24.6369216Z + HAS_NVIDIA_DRIVER=1 2024-06-01T03:35:24.6370146Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-01T03:35:24.6371003Z + set -e 2024-06-01T03:35:24.6371308Z + '[' 1 -eq 0 ']' 2024-06-01T03:35:24.6371903Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-01T03:35:24.6372608Z + post_install_nvidia_driver_common 2024-06-01T03:35:24.6373023Z + sudo modprobe nvidia 2024-06-01T03:35:24.6462273Z + echo 'After installing NVIDIA driver' 2024-06-01T03:35:24.6462927Z + lspci 2024-06-01T03:35:24.6463702Z After installing NVIDIA driver 2024-06-01T03:35:24.6552396Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-01T03:35:24.6553508Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-01T03:35:24.6554828Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-01T03:35:24.6555966Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-01T03:35:24.6557116Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-01T03:35:24.6558351Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-01T03:35:24.6559383Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-01T03:35:24.6560541Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-01T03:35:24.6561438Z + lsmod 2024-06-01T03:35:24.6567697Z Module Size Used by 2024-06-01T03:35:24.6568283Z nvidia_uvm 4599808 0 2024-06-01T03:35:24.6568854Z nvidia 53989376 1 nvidia_uvm 2024-06-01T03:35:24.6569444Z drm 421888 1 nvidia 2024-06-01T03:35:24.6570032Z i2c_core 77824 2 nvidia,drm 2024-06-01T03:35:24.6570638Z backlight 16384 0 2024-06-01T03:35:24.6571097Z xt_conntrack 16384 1 2024-06-01T03:35:24.6571516Z ipt_MASQUERADE 16384 1 2024-06-01T03:35:24.6571957Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-01T03:35:24.6572445Z nf_conntrack_netlink 49152 0 2024-06-01T03:35:24.6572919Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-01T03:35:24.6573396Z xfrm_user 45056 1 2024-06-01T03:35:24.6573793Z xfrm_algo 16384 1 xfrm_user 2024-06-01T03:35:24.6574227Z iptable_nat 16384 1 2024-06-01T03:35:24.6574620Z nf_conntrack_ipv4 16384 3 2024-06-01T03:35:24.6575060Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-01T03:35:24.6575554Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-01T03:35:24.6576272Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-01T03:35:24.6577562Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-01T03:35:24.6578736Z xt_addrtype 16384 2 2024-06-01T03:35:24.6579233Z iptable_filter 16384 1 2024-06-01T03:35:24.6579631Z br_netfilter 24576 0 2024-06-01T03:35:24.6580055Z bridge 172032 1 br_netfilter 2024-06-01T03:35:24.6580511Z stp 16384 1 bridge 2024-06-01T03:35:24.6580942Z llc 16384 2 bridge,stp 2024-06-01T03:35:24.6581369Z overlay 86016 0 2024-06-01T03:35:24.6581749Z sunrpc 393216 1 2024-06-01T03:35:24.6582119Z dm_mirror 28672 0 2024-06-01T03:35:24.6582527Z dm_region_hash 20480 1 dm_mirror 2024-06-01T03:35:24.6583030Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-01T03:35:24.6583597Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-01T03:35:24.6584060Z dax 69632 1 dm_mod 2024-06-01T03:35:24.6584556Z crc32_pclmul 16384 0 2024-06-01T03:35:24.6584948Z ghash_clmulni_intel 16384 0 2024-06-01T03:35:24.6585345Z pcbc 16384 0 2024-06-01T03:35:24.6585715Z aesni_intel 188416 0 2024-06-01T03:35:24.6586124Z aes_x86_64 20480 1 aesni_intel 2024-06-01T03:35:24.6586585Z crypto_simd 16384 1 aesni_intel 2024-06-01T03:35:24.6587050Z glue_helper 16384 1 aesni_intel 2024-06-01T03:35:24.6587625Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-01T03:35:24.6588181Z mousedev 24576 0 2024-06-01T03:35:24.6588557Z evdev 20480 3 2024-06-01T03:35:24.6588928Z psmouse 32768 0 2024-06-01T03:35:24.6589306Z button 16384 0 2024-06-01T03:35:24.6589677Z ena 139264 0 2024-06-01T03:35:24.6590373Z ptp 20480 1 ena 2024-06-01T03:35:24.6591039Z pps_core 20480 1 ptp 2024-06-01T03:35:24.6591469Z crc32c_intel 24576 0 2024-06-01T03:35:24.6591849Z autofs4 49152 2 2024-06-01T03:35:24.6592221Z + modinfo nvidia 2024-06-01T03:35:24.6592898Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-01T03:35:24.6593607Z alias: char-major-195-* 2024-06-01T03:35:24.6593997Z version: 550.54.15 2024-06-01T03:35:24.6594425Z supported: external 2024-06-01T03:35:24.6594766Z license: NVIDIA 2024-06-01T03:35:24.6595154Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-01T03:35:24.6595650Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-01T03:35:24.6596118Z srcversion: 833721318DA517F0C2FEC97 2024-06-01T03:35:24.6596588Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-01T03:35:24.6597098Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-01T03:35:24.6597605Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-01T03:35:24.6598096Z depends: i2c-core,drm 2024-06-01T03:35:24.6598464Z retpoline: Y 2024-06-01T03:35:24.6598782Z name: nvidia 2024-06-01T03:35:24.6599344Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-01T03:35:24.6600006Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-01T03:35:24.6600670Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-01T03:35:24.6601283Z parm: NVreg_ResmanDebugLevel:int 2024-06-01T03:35:24.6601737Z parm: NVreg_RmLogonRC:int 2024-06-01T03:35:24.6602176Z parm: NVreg_ModifyDeviceFiles:int 2024-06-01T03:35:24.6602644Z parm: NVreg_DeviceFileUID:int 2024-06-01T03:35:24.6603254Z parm: NVreg_DeviceFileGID:int 2024-06-01T03:35:24.6603704Z parm: NVreg_DeviceFileMode:int 2024-06-01T03:35:24.6604242Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-01T03:35:24.6604821Z parm: NVreg_UsePageAttributeTable:int 2024-06-01T03:35:24.6605319Z parm: NVreg_EnablePCIeGen3:int 2024-06-01T03:35:24.6605762Z parm: NVreg_EnableMSI:int 2024-06-01T03:35:24.6606183Z parm: NVreg_TCEBypassMode:int 2024-06-01T03:35:24.6606653Z parm: NVreg_EnableStreamMemOPs:int 2024-06-01T03:35:24.6607196Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-01T03:35:24.6607783Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-01T03:35:24.6608370Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-01T03:35:24.6609000Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-01T03:35:24.6609614Z parm: NVreg_DynamicPowerManagement:int 2024-06-01T03:35:24.6610239Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-01T03:35:24.6610849Z parm: NVreg_EnableGpuFirmware:int 2024-06-01T03:35:24.6611355Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-01T03:35:24.6611912Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-01T03:35:24.6612474Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-01T03:35:24.6612982Z parm: NVreg_MemoryPoolSize:int 2024-06-01T03:35:24.6613467Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-01T03:35:24.6613962Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-01T03:35:24.6614441Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-01T03:35:24.6614900Z parm: NVreg_NvLinkDisable:int 2024-06-01T03:35:24.6615425Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-01T03:35:24.6615968Z parm: NVreg_RegisterPCIDriver:int 2024-06-01T03:35:24.6616451Z parm: NVreg_EnableResizableBar:int 2024-06-01T03:35:24.6616953Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-01T03:35:24.6617463Z parm: NVreg_EnableNonblockingOpen:int 2024-06-01T03:35:24.6617970Z parm: NVreg_RegistryDwords:charp 2024-06-01T03:35:24.6618475Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-01T03:35:24.6619034Z parm: NVreg_RmMsg:charp 2024-06-01T03:35:24.6619463Z parm: NVreg_GpuBlacklist:charp 2024-06-01T03:35:24.6619940Z parm: NVreg_TemporaryFilePath:charp 2024-06-01T03:35:24.6620411Z parm: NVreg_ExcludedGpus:charp 2024-06-01T03:35:24.6620906Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-01T03:35:24.6621418Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-01T03:35:24.6621898Z parm: NVreg_ImexChannelCount:int 2024-06-01T03:35:24.6622421Z parm: rm_firmware_active:charp 2024-06-01T03:35:24.6622828Z + set +e 2024-06-01T03:35:24.6623130Z + nvidia-smi 2024-06-01T03:35:26.2244216Z Sat Jun 1 03:35:26 2024 2024-06-01T03:35:26.2245178Z +-----------------------------------------------------------------------------------------+ 2024-06-01T03:35:26.2246270Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-01T03:35:26.2247138Z |-----------------------------------------+------------------------+----------------------+ 2024-06-01T03:35:26.2247965Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-01T03:35:26.2248873Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-01T03:35:26.2249592Z | | | MIG M. | 2024-06-01T03:35:26.2250142Z |=========================================+========================+======================| 2024-06-01T03:35:26.2340311Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-01T03:35:26.2341140Z | 0% 28C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-01T03:35:26.2341804Z | | | N/A | 2024-06-01T03:35:26.2342641Z +-----------------------------------------+------------------------+----------------------+ 2024-06-01T03:35:26.2343486Z 2024-06-01T03:35:26.2344177Z +-----------------------------------------------------------------------------------------+ 2024-06-01T03:35:26.2344836Z | Processes: | 2024-06-01T03:35:26.2345674Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-01T03:35:26.2346539Z | ID ID Usage | 2024-06-01T03:35:26.2347132Z |=========================================================================================| 2024-06-01T03:35:26.2347813Z | No running processes found | 2024-06-01T03:35:26.2348612Z +-----------------------------------------------------------------------------------------+ 2024-06-01T03:35:26.8593327Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-01T03:35:28.4283610Z NVIDIA A10G 2024-06-01T03:35:28.8723514Z + NVIDIA_SMI_STATUS=0 2024-06-01T03:35:28.8724084Z + '[' 0 -eq 0 ']' 2024-06-01T03:35:28.8724603Z + echo 'INFO: Ignoring allowed status 0' 2024-06-01T03:35:28.8725221Z + set -e 2024-06-01T03:35:28.8725672Z INFO: Ignoring allowed status 0 2024-06-01T03:35:28.8727960Z == Installing nvidia container toolkit for amzn2 == 2024-06-01T03:35:28.8730538Z + sudo yum install -y yum-utils 2024-06-01T03:35:29.1547908Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-01T03:35:30.6484124Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-01T03:35:30.6484847Z Nothing to do 2024-06-01T03:35:30.7999701Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-01T03:35:31.1024049Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-01T03:35:31.1273026Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-01T03:35:31.1274414Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-01T03:35:31.1275407Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-01T03:35:31.1380144Z + sudo yum install -y nvidia-docker2 2024-06-01T03:35:31.4300416Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-01T03:35:32.8934853Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-01T03:35:32.8935693Z Nothing to do 2024-06-01T03:35:33.0415647Z + sudo systemctl restart docker 2024-06-01T03:36:26.2282697Z Command completed after 1 attempt(s). 2024-06-01T03:36:26.2341984Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-01T03:36:26.2342796Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-01T03:36:26.2343520Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-01T03:36:26.2344206Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-01T03:36:26.2351839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:26.2352355Z env: 2024-06-01T03:36:26.2352637Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:26.2353096Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:26.2353582Z ##[endgroup] 2024-06-01T03:36:26.5923877Z Defaulting to user installation because normal site-packages is not writeable 2024-06-01T03:36:26.9990599Z Collecting psutil==5.9.1 2024-06-01T03:36:27.0208623Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-06-01T03:36:27.0681041Z Collecting nvidia-ml-py==11.525.84 2024-06-01T03:36:27.0726099Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-06-01T03:36:27.1117680Z Installing collected packages: psutil, nvidia-ml-py 2024-06-01T03:36:27.2675144Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-06-01T03:36:27.4572673Z Prepare all required actions 2024-06-01T03:36:27.4573170Z Getting action download info 2024-06-01T03:36:27.5725491Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-01T03:36:27.7276908Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-01T03:36:27.8319984Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-01T03:36:27.8320502Z with: 2024-06-01T03:36:27.8320849Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:36:27.8321318Z s3-bucket: gha-artifacts 2024-06-01T03:36:27.8321666Z env: 2024-06-01T03:36:27.8321952Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:27.8322406Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:27.8323051Z ##[endgroup] 2024-06-01T03:36:27.8369318Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-01T03:36:27.8369766Z with: 2024-06-01T03:36:27.8370141Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:36:27.8370609Z s3-bucket: gha-artifacts 2024-06-01T03:36:27.8370955Z region: us-east-1 2024-06-01T03:36:27.8371254Z env: 2024-06-01T03:36:27.8371538Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:27.8371990Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:27.8372479Z ##[endgroup] 2024-06-01T03:36:28.2503702Z (node:4969) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-01T03:36:28.2504460Z 2024-06-01T03:36:28.2504728Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-01T03:36:28.2505482Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-01T03:36:28.2506386Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-01T03:36:28.3369358Z Found 1 objects with prefix pytorch/pytorch/9326485603/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-06-01T03:36:28.3370506Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-01T03:36:35.8359346Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-01T03:36:35.8365781Z Artifact download has finished successfully 2024-06-01T03:36:35.8521618Z ##[group]Run unzip -o artifacts.zip 2024-06-01T03:36:35.8522074Z unzip -o artifacts.zip 2024-06-01T03:36:35.8529722Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:35.8530235Z env: 2024-06-01T03:36:35.8530625Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:35.8531100Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:35.8533098Z ##[endgroup] 2024-06-01T03:36:35.8582560Z Archive: artifacts.zip 2024-06-01T03:36:35.8584134Z creating: dist/ 2024-06-01T03:36:37.8274183Z inflating: dist/torch-2.4.0a0+gitde352ff-cp310-cp310-linux_x86_64.whl 2024-06-01T03:36:37.8274911Z creating: build/custom_test_artifacts/ 2024-06-01T03:36:37.8275517Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-01T03:36:37.8276269Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-01T03:36:37.8277132Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-01T03:36:37.8281442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-01T03:36:37.8282438Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-01T03:36:37.8283545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-01T03:36:37.8284594Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-01T03:36:37.8285611Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-01T03:36:37.8286757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-01T03:36:37.8288103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-01T03:36:37.8289215Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-01T03:36:37.8290252Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-01T03:36:37.8291448Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-01T03:36:37.8292652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-01T03:36:37.8293841Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-01T03:36:37.8295023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-01T03:36:37.8296339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-01T03:36:37.8297560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-01T03:36:37.8298625Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-01T03:36:37.8299674Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-01T03:36:37.8335643Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-01T03:36:37.8376180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-01T03:36:37.8377717Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-01T03:36:37.8422527Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-01T03:36:37.8425551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-01T03:36:37.8428588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-01T03:36:37.8430715Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-01T03:36:37.8432201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-01T03:36:37.8433841Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-01T03:36:37.8435296Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-01T03:36:37.8436741Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-01T03:36:37.8438172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-01T03:36:37.8439507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-01T03:36:37.8440785Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-01T03:36:37.8442058Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-01T03:36:37.8444721Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-01T03:36:37.8445970Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-01T03:36:37.8447241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-01T03:36:37.8502035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-01T03:36:37.8571645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-01T03:36:37.8572905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-01T03:36:37.8573950Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-01T03:36:37.8574857Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-01T03:36:37.8575804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-01T03:36:37.8576785Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-01T03:36:37.8577868Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-01T03:36:37.8579088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-01T03:36:37.8580257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-01T03:36:37.8581346Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-01T03:36:37.8582483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-01T03:36:37.8583630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-01T03:36:37.8584774Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-01T03:36:37.8585901Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-01T03:36:37.8587022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-01T03:36:37.8596697Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-01T03:36:37.8724708Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-01T03:36:37.8725756Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-01T03:36:37.8726898Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-01T03:36:37.8728170Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-01T03:36:37.8729514Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-01T03:36:37.8730658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-01T03:36:37.8731833Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-01T03:36:37.8733016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-01T03:36:37.8734201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-01T03:36:37.8735376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-01T03:36:37.8736538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-01T03:36:37.8748223Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-01T03:36:37.8827216Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-01T03:36:37.8828502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-01T03:36:37.8830807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-01T03:36:37.8833309Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-01T03:36:37.8835377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-01T03:36:37.8837205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-01T03:36:37.8838833Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-01T03:36:37.8839725Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-01T03:36:37.8840528Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-01T03:36:37.8841350Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-01T03:36:37.8942177Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-01T03:36:37.9001878Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-01T03:36:37.9002941Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-01T03:36:37.9003666Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-01T03:36:37.9004512Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-01T03:36:37.9009431Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-01T03:36:37.9010618Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-01T03:36:37.9011728Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-01T03:36:37.9012970Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-01T03:36:37.9014168Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-01T03:36:37.9015375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-01T03:36:37.9016813Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-01T03:36:37.9018074Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-01T03:36:37.9019400Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-01T03:36:37.9020824Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-01T03:36:37.9022251Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-01T03:36:37.9023556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-01T03:36:37.9024744Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-01T03:36:37.9025959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-01T03:36:37.9027172Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-01T03:36:37.9028257Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-01T03:36:37.9029297Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-01T03:36:37.9061683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-01T03:36:37.9100860Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-01T03:36:37.9102375Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-01T03:36:37.9145915Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-01T03:36:37.9147941Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-01T03:36:37.9150175Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-01T03:36:37.9152262Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-01T03:36:37.9154044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-01T03:36:37.9155549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-01T03:36:37.9157002Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-01T03:36:37.9158428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-01T03:36:37.9159821Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-01T03:36:37.9161142Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-01T03:36:37.9162412Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-01T03:36:37.9163829Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-01T03:36:37.9165087Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-01T03:36:37.9166313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-01T03:36:37.9167561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-01T03:36:37.9226653Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-01T03:36:37.9296270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-01T03:36:37.9297813Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-01T03:36:37.9299080Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-01T03:36:37.9300035Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-01T03:36:37.9301262Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-01T03:36:37.9302243Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-01T03:36:37.9303327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-01T03:36:37.9304557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-01T03:36:37.9305755Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-01T03:36:37.9306859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-01T03:36:37.9308001Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-01T03:36:37.9309135Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-01T03:36:37.9310538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-01T03:36:37.9311692Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-01T03:36:37.9312834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-01T03:36:37.9321514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-01T03:36:37.9382723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-01T03:36:37.9384200Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-01T03:36:37.9385453Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-01T03:36:37.9386565Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-01T03:36:37.9387548Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-01T03:36:37.9388617Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-01T03:36:37.9389552Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-01T03:36:37.9390656Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-01T03:36:37.9391452Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-01T03:36:37.9392276Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-01T03:36:37.9436665Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-01T03:36:37.9437631Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-01T03:36:37.9438416Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-01T03:36:37.9439326Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-01T03:36:37.9445891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-01T03:36:37.9447057Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-01T03:36:37.9448213Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-01T03:36:37.9449587Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-01T03:36:37.9450794Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-01T03:36:37.9452135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-01T03:36:37.9453650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-01T03:36:37.9454979Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-01T03:36:37.9456347Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-01T03:36:37.9458059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-01T03:36:37.9459414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-01T03:36:37.9460656Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-01T03:36:37.9461900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-01T03:36:37.9463166Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-01T03:36:37.9464421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-01T03:36:37.9465558Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-01T03:36:37.9466663Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-01T03:36:37.9498548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-01T03:36:37.9537521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-01T03:36:37.9539528Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-01T03:36:37.9582759Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-01T03:36:37.9584776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-01T03:36:37.9586709Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-01T03:36:37.9588887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-01T03:36:37.9591336Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-01T03:36:37.9593028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-01T03:36:37.9594583Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-01T03:36:37.9596089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-01T03:36:37.9597549Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-01T03:36:37.9598957Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-01T03:36:37.9600298Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-01T03:36:37.9601626Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-01T03:36:37.9603032Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-01T03:36:37.9604324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-01T03:36:37.9605632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-01T03:36:37.9662351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-01T03:36:37.9733279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-01T03:36:37.9734610Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-01T03:36:37.9735720Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-01T03:36:37.9736691Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-01T03:36:37.9737704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-01T03:36:37.9738775Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-01T03:36:37.9739965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-01T03:36:37.9741320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-01T03:36:37.9742600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-01T03:36:37.9743813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-01T03:36:37.9745059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-01T03:36:37.9746448Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-01T03:36:37.9747707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-01T03:36:37.9748941Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-01T03:36:37.9751524Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-01T03:36:37.9753001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-01T03:36:37.9863509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-01T03:36:37.9864748Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-01T03:36:37.9865976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-01T03:36:37.9867354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-01T03:36:37.9868671Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-01T03:36:37.9870045Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-01T03:36:37.9871369Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-01T03:36:37.9872678Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-01T03:36:37.9873987Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-01T03:36:37.9875296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-01T03:36:37.9876590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-01T03:36:37.9887852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-01T03:36:37.9940378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-01T03:36:37.9941773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-01T03:36:37.9944379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-01T03:36:37.9945455Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-01T03:36:37.9946453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-01T03:36:37.9947459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-01T03:36:37.9948455Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-01T03:36:37.9949392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-01T03:36:37.9950418Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-01T03:36:37.9951309Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-01T03:36:38.0047044Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-01T03:36:38.0088020Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-01T03:36:38.0088682Z creating: build/lib/ 2024-06-01T03:36:38.0172932Z inflating: build/lib/libprotobuf-lite.a 2024-06-01T03:36:38.0611628Z inflating: build/lib/libprotobuf.a 2024-06-01T03:36:38.0620446Z inflating: build/lib/libpthreadpool.a 2024-06-01T03:36:38.0628006Z inflating: build/lib/libcpuinfo.a 2024-06-01T03:36:38.0635738Z inflating: build/lib/libcpuinfo_internals.a 2024-06-01T03:36:38.0636266Z inflating: build/lib/libclog.a 2024-06-01T03:36:38.0653609Z inflating: build/lib/libnnpack.a 2024-06-01T03:36:38.0655414Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-01T03:36:38.0717864Z inflating: build/lib/libgtest.a 2024-06-01T03:36:38.0789202Z inflating: build/lib/libbenchmark.a 2024-06-01T03:36:38.0850733Z inflating: build/lib/libasmjit.a 2024-06-01T03:36:38.0857798Z inflating: build/lib/libittnotify.a 2024-06-01T03:36:38.0884486Z inflating: build/lib/libtensorpipe_uv.a 2024-06-01T03:36:38.1006754Z inflating: build/lib/libgloo.a 2024-06-01T03:36:38.1027503Z inflating: build/lib/libfmt.a 2024-06-01T03:36:38.1115273Z inflating: build/lib/libc10.so 2024-06-01T03:36:38.1116672Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-01T03:36:38.1117456Z inflating: build/lib/libfoxi_loader.a 2024-06-01T03:36:38.1118968Z inflating: build/lib/libtorch_global_deps.so 2024-06-01T03:36:38.1137551Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-01T03:36:38.1623760Z inflating: build/lib/libprotoc.a 2024-06-01T03:36:38.1640239Z inflating: build/lib/libgmock.a 2024-06-01T03:36:38.1640730Z inflating: build/lib/libbenchmark_main.a 2024-06-01T03:36:38.1641341Z inflating: build/lib/libgtest_main.a 2024-06-01T03:36:38.2010183Z inflating: build/lib/libgloo_cuda.a 2024-06-01T03:36:39.1735784Z inflating: build/lib/libdnnl.a 2024-06-01T03:36:39.2286473Z inflating: build/lib/libtensorpipe.a 2024-06-01T03:36:39.2339210Z inflating: build/lib/libc10_cuda.so 2024-06-01T03:36:39.2340215Z inflating: build/lib/libgmock_main.a 2024-06-01T03:36:39.3542361Z inflating: build/lib/libfbgemm.a 2024-06-01T03:36:39.4037594Z inflating: build/lib/libkineto.a 2024-06-01T03:36:39.4282114Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-01T03:36:39.4319204Z inflating: build/lib/libcaffe2_protos.a 2024-06-01T03:36:39.4359665Z inflating: build/lib/libonnx_proto.a 2024-06-01T03:36:39.4537553Z inflating: build/lib/libXNNPACK.a 2024-06-01T03:36:39.5211349Z inflating: build/lib/libonnx.a 2024-06-01T03:36:41.8464162Z inflating: build/lib/libtorch_cpu.so 2024-06-01T03:36:41.8468234Z inflating: build/lib/libunbox_lib.a 2024-06-01T03:36:41.8471754Z inflating: build/lib/libshm.so 2024-06-01T03:36:43.8214048Z inflating: build/lib/libtorch_cuda.so 2024-06-01T03:36:43.8215055Z inflating: build/lib/libtorch.so 2024-06-01T03:36:44.6730729Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-01T03:36:44.6732739Z inflating: build/lib/libc10d_cuda_test.so 2024-06-01T03:36:44.8584428Z inflating: build/lib/libtorch_python.so 2024-06-01T03:36:44.8649482Z inflating: build/lib/libtorchbind_test.so 2024-06-01T03:36:44.8669315Z inflating: build/lib/libjitbackend_test.so 2024-06-01T03:36:44.8694012Z inflating: build/lib/libbackend_with_compiler.so 2024-06-01T03:36:44.8727393Z inflating: build/lib/libnnapi_backend.so 2024-06-01T03:36:44.8727956Z creating: build/bin/ 2024-06-01T03:36:44.8776360Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-01T03:36:44.8826630Z inflating: build/bin/c10_DeviceGuard_test 2024-06-01T03:36:44.8875386Z inflating: build/bin/c10_Device_test 2024-06-01T03:36:44.8931927Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-01T03:36:44.8985695Z inflating: build/bin/c10_Scalar_test 2024-06-01T03:36:44.9032460Z inflating: build/bin/c10_StreamGuard_test 2024-06-01T03:36:44.9081611Z inflating: build/bin/c10_SymInt_test 2024-06-01T03:36:44.9134280Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-01T03:36:44.9187929Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-01T03:36:44.9242409Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-01T03:36:44.9311240Z inflating: build/bin/c10_cow_test 2024-06-01T03:36:44.9362396Z inflating: build/bin/c10_Bitset_test 2024-06-01T03:36:44.9409946Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-01T03:36:44.9457830Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-01T03:36:44.9507528Z inflating: build/bin/c10_Half_test 2024-06-01T03:36:44.9561343Z inflating: build/bin/c10_LeftRight_test 2024-06-01T03:36:44.9614624Z inflating: build/bin/c10_Metaprogramming_test 2024-06-01T03:36:44.9662832Z inflating: build/bin/c10_Synchronized_test 2024-06-01T03:36:44.9716645Z inflating: build/bin/c10_ThreadLocal_test 2024-06-01T03:36:44.9766517Z inflating: build/bin/c10_TypeIndex_test 2024-06-01T03:36:44.9815744Z inflating: build/bin/c10_TypeList_test 2024-06-01T03:36:44.9862827Z inflating: build/bin/c10_TypeTraits_test 2024-06-01T03:36:44.9913587Z inflating: build/bin/c10_accumulate_test 2024-06-01T03:36:44.9967907Z inflating: build/bin/c10_bfloat16_test 2024-06-01T03:36:45.0016827Z inflating: build/bin/c10_bit_cast_test 2024-06-01T03:36:45.0071301Z inflating: build/bin/c10_complex_math_test 2024-06-01T03:36:45.0125057Z inflating: build/bin/c10_complex_test 2024-06-01T03:36:45.0175900Z inflating: build/bin/c10_exception_test 2024-06-01T03:36:45.0225099Z inflating: build/bin/c10_flags_test 2024-06-01T03:36:45.0273001Z inflating: build/bin/c10_generic_math_test 2024-06-01T03:36:45.0431306Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-01T03:36:45.0480413Z inflating: build/bin/c10_irange_test 2024-06-01T03:36:45.0532144Z inflating: build/bin/c10_lazy_test 2024-06-01T03:36:45.0587543Z inflating: build/bin/c10_logging_test 2024-06-01T03:36:45.0660138Z inflating: build/bin/c10_optional_test 2024-06-01T03:36:45.0720626Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-01T03:36:45.0773790Z inflating: build/bin/c10_registry_test 2024-06-01T03:36:45.0918794Z inflating: build/bin/c10_small_vector_test 2024-06-01T03:36:45.0968750Z inflating: build/bin/c10_ssize_test 2024-06-01T03:36:45.1018914Z inflating: build/bin/c10_string_util_test 2024-06-01T03:36:45.1076001Z inflating: build/bin/c10_string_view_test 2024-06-01T03:36:45.1124459Z inflating: build/bin/c10_tempfile_test 2024-06-01T03:36:45.1171482Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-01T03:36:45.1226114Z inflating: build/bin/c10_typeid_test 2024-06-01T03:36:45.1658271Z inflating: build/bin/protoc-3.13.0.0 2024-06-01T03:36:45.2093370Z inflating: build/bin/protoc 2024-06-01T03:36:45.2144827Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-01T03:36:45.2196071Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-01T03:36:45.2246484Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-01T03:36:45.2296484Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-01T03:36:45.2347226Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-01T03:36:45.2398529Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-01T03:36:45.2446020Z inflating: build/bin/c10_cuda_CUDATest 2024-06-01T03:36:45.2497116Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-01T03:36:45.2814842Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-01T03:36:45.3150666Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-01T03:36:45.3494406Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-01T03:36:45.3545609Z inflating: build/bin/FileStoreTest 2024-06-01T03:36:45.3600075Z inflating: build/bin/TCPStoreTest 2024-06-01T03:36:45.3651324Z inflating: build/bin/HashStoreTest 2024-06-01T03:36:45.3664238Z inflating: build/bin/ProcessGroupMPITest 2024-06-01T03:36:45.3716272Z inflating: build/bin/test_edge_op_registration 2024-06-01T03:36:45.3720238Z inflating: build/bin/torch_shm_manager 2024-06-01T03:36:45.3722933Z inflating: build/bin/example_allreduce 2024-06-01T03:36:45.3775976Z inflating: build/bin/test_dist_autograd 2024-06-01T03:36:45.3843015Z inflating: build/bin/test_cpp_rpc 2024-06-01T03:36:45.3845430Z inflating: build/bin/parallel_benchmark 2024-06-01T03:36:45.3910579Z inflating: build/bin/test_mobile_nnc 2024-06-01T03:36:45.3918687Z inflating: build/bin/aot_model_compiler_test 2024-06-01T03:36:45.4253628Z inflating: build/bin/test_lazy 2024-06-01T03:36:45.5400030Z inflating: build/bin/test_api 2024-06-01T03:36:45.5471363Z inflating: build/bin/Dict_test 2024-06-01T03:36:45.5521624Z inflating: build/bin/Dimname_test 2024-06-01T03:36:45.5584357Z inflating: build/bin/MaybeOwned_test 2024-06-01T03:36:45.5639237Z inflating: build/bin/NamedTensor_test 2024-06-01T03:36:45.5696426Z inflating: build/bin/apply_utils_test 2024-06-01T03:36:45.5753380Z inflating: build/bin/atest 2024-06-01T03:36:45.5814418Z inflating: build/bin/basic 2024-06-01T03:36:45.5867965Z inflating: build/bin/broadcast_test 2024-06-01T03:36:45.5916699Z inflating: build/bin/cpu_allocator_test 2024-06-01T03:36:45.5972993Z inflating: build/bin/cpu_generator_test 2024-06-01T03:36:45.6024815Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-01T03:36:45.6114500Z inflating: build/bin/cpu_rng_test 2024-06-01T03:36:45.6162890Z inflating: build/bin/dispatch_key_set_test 2024-06-01T03:36:45.6211406Z inflating: build/bin/dlconvertor_test 2024-06-01T03:36:45.6268931Z inflating: build/bin/extension_backend_test 2024-06-01T03:36:45.6322341Z inflating: build/bin/half_test 2024-06-01T03:36:45.6415242Z inflating: build/bin/ivalue_test 2024-06-01T03:36:45.6463435Z inflating: build/bin/lazy_tensor_test 2024-06-01T03:36:45.6516099Z inflating: build/bin/math_kernel_test 2024-06-01T03:36:45.6568627Z inflating: build/bin/memory_format_test 2024-06-01T03:36:45.6619857Z inflating: build/bin/memory_overlapping_test 2024-06-01T03:36:45.6671283Z inflating: build/bin/mobile_memory_cleanup 2024-06-01T03:36:45.6725502Z inflating: build/bin/native_test 2024-06-01T03:36:45.6774203Z inflating: build/bin/operator_name_test 2024-06-01T03:36:45.6824490Z inflating: build/bin/operators_test 2024-06-01T03:36:45.6874260Z inflating: build/bin/packedtensoraccessor_test 2024-06-01T03:36:45.6939374Z inflating: build/bin/pow_test 2024-06-01T03:36:45.6995751Z inflating: build/bin/quantized_test 2024-06-01T03:36:45.7043864Z inflating: build/bin/reduce_ops_test 2024-06-01T03:36:45.7093506Z inflating: build/bin/reportMemoryUsage_test 2024-06-01T03:36:45.7148801Z inflating: build/bin/scalar_tensor_test 2024-06-01T03:36:45.7205157Z inflating: build/bin/scalar_test 2024-06-01T03:36:45.7255235Z inflating: build/bin/StorageUtils_test 2024-06-01T03:36:45.7305998Z inflating: build/bin/stride_properties_test 2024-06-01T03:36:45.7382309Z inflating: build/bin/tensor_iterator_test 2024-06-01T03:36:45.7434508Z inflating: build/bin/test_parallel 2024-06-01T03:36:45.7437178Z inflating: build/bin/thread_init_test 2024-06-01T03:36:45.7491150Z inflating: build/bin/type_ptr_test 2024-06-01T03:36:45.7549514Z inflating: build/bin/type_test 2024-06-01T03:36:45.7599535Z inflating: build/bin/undefined_tensor_test 2024-06-01T03:36:45.7600851Z inflating: build/bin/verify_api_visibility 2024-06-01T03:36:45.7668012Z inflating: build/bin/legacy_vmap_test 2024-06-01T03:36:45.7718526Z inflating: build/bin/weakref_test 2024-06-01T03:36:45.7769246Z inflating: build/bin/wrapdim_test 2024-06-01T03:36:45.7819253Z inflating: build/bin/xla_tensor_test 2024-06-01T03:36:45.7877937Z inflating: build/bin/IListRef_test 2024-06-01T03:36:45.7980568Z inflating: build/bin/List_test 2024-06-01T03:36:45.8098184Z inflating: build/bin/kernel_function_legacy_test 2024-06-01T03:36:45.8163432Z inflating: build/bin/KernelFunction_test 2024-06-01T03:36:45.8256839Z inflating: build/bin/kernel_function_test 2024-06-01T03:36:45.8381096Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-01T03:36:45.8480981Z inflating: build/bin/kernel_lambda_test 2024-06-01T03:36:45.8540046Z inflating: build/bin/kernel_stackbased_test 2024-06-01T03:36:45.8633697Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-01T03:36:45.8682482Z inflating: build/bin/CppSignature_test 2024-06-01T03:36:45.8729851Z inflating: build/bin/op_allowlist_test 2024-06-01T03:36:45.8783957Z inflating: build/bin/backend_fallback_test 2024-06-01T03:36:45.9075348Z inflating: build/bin/op_registration_test 2024-06-01T03:36:45.9138496Z inflating: build/bin/inline_container_test 2024-06-01T03:36:45.9189784Z inflating: build/bin/cuda_apply_test 2024-06-01T03:36:45.9239167Z inflating: build/bin/cuda_allocator_test 2024-06-01T03:36:45.9292612Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-01T03:36:45.9349798Z inflating: build/bin/cuda_atomic_ops_test 2024-06-01T03:36:45.9417064Z inflating: build/bin/cuda_complex_math_test 2024-06-01T03:36:45.9474911Z inflating: build/bin/cuda_complex_test 2024-06-01T03:36:45.9522426Z inflating: build/bin/cuda_device_test 2024-06-01T03:36:45.9579042Z inflating: build/bin/cuda_cub_test 2024-06-01T03:36:45.9628209Z inflating: build/bin/cuda_dlconvertor_test 2024-06-01T03:36:45.9691031Z inflating: build/bin/cuda_distributions_test 2024-06-01T03:36:45.9746691Z inflating: build/bin/cuda_generator_test 2024-06-01T03:36:45.9795344Z inflating: build/bin/cuda_half_test 2024-06-01T03:36:45.9844678Z inflating: build/bin/cuda_integer_divider_test 2024-06-01T03:36:45.9892386Z inflating: build/bin/cuda_optional_test 2024-06-01T03:36:45.9942574Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-01T03:36:45.9993588Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-01T03:36:46.0042064Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-01T03:36:46.0100681Z inflating: build/bin/cuda_stream_test 2024-06-01T03:36:46.0148326Z inflating: build/bin/cuda_cudnn_test 2024-06-01T03:36:46.0199669Z inflating: build/bin/cuda_vectorized_test 2024-06-01T03:36:46.0213537Z inflating: build/bin/tutorial_tensorexpr 2024-06-01T03:36:46.0277164Z inflating: build/bin/ProcessGroupGlooTest 2024-06-01T03:36:46.0332403Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-01T03:36:46.0393367Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-01T03:36:46.0453370Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-01T03:36:46.1259122Z inflating: build/bin/test_tensorexpr 2024-06-01T03:36:46.1818840Z inflating: build/bin/test_jit 2024-06-01T03:36:46.1819308Z creating: .additional_ci_files/ 2024-06-01T03:36:46.1866836Z inflating: .additional_ci_files/test-times.json 2024-06-01T03:36:46.2051841Z inflating: .additional_ci_files/test-class-times.json 2024-06-01T03:36:46.2079590Z ##[group]Run rm artifacts.zip 2024-06-01T03:36:46.2080002Z rm artifacts.zip 2024-06-01T03:36:46.2087453Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:46.2087969Z env: 2024-06-01T03:36:46.2088242Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.2088705Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.2089202Z ##[endgroup] 2024-06-01T03:36:46.2797260Z ##[group]Run df -H 2024-06-01T03:36:46.2797591Z df -H 2024-06-01T03:36:46.2805107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:46.2805624Z env: 2024-06-01T03:36:46.2805902Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.2806385Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.2806889Z ##[endgroup] 2024-06-01T03:36:46.2832524Z Filesystem Size Used Avail Use% Mounted on 2024-06-01T03:36:46.2833098Z devtmpfs 34G 0 34G 0% /dev 2024-06-01T03:36:46.2833616Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-01T03:36:46.2834337Z tmpfs 34G 455k 34G 1% /run 2024-06-01T03:36:46.2834843Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-01T03:36:46.2835474Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-01T03:36:46.2886294Z Prepare all required actions 2024-06-01T03:36:46.2886733Z Getting action download info 2024-06-01T03:36:46.3961157Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-01T03:36:46.3961631Z with: 2024-06-01T03:36:46.3961887Z env: 2024-06-01T03:36:46.3962169Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.3962712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.3963214Z ##[endgroup] 2024-06-01T03:36:46.4010122Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-01T03:36:46.4010581Z with: 2024-06-01T03:36:46.4010844Z name: td_results 2024-06-01T03:36:46.4011164Z s3-bucket: gha-artifacts 2024-06-01T03:36:46.4011514Z region: us-east-1 2024-06-01T03:36:46.4011814Z env: 2024-06-01T03:36:46.4012082Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.4012539Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.4013028Z ##[endgroup] 2024-06-01T03:36:46.8210981Z (node:4998) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-01T03:36:46.8211903Z 2024-06-01T03:36:46.8212179Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-01T03:36:46.8212941Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-01T03:36:46.8213804Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-01T03:36:46.8994016Z Found 0 objects with prefix pytorch/pytorch/9326485603/td_results/ 2024-06-01T03:36:46.9000447Z Artifact download has finished successfully 2024-06-01T03:36:46.9128147Z ##[group]Run mkdir -p .additional_ci_files 2024-06-01T03:36:46.9128655Z mkdir -p .additional_ci_files 2024-06-01T03:36:46.9129245Z mv td_results.json .additional_ci_files/td_results.json 2024-06-01T03:36:46.9136914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:46.9137411Z env: 2024-06-01T03:36:46.9137692Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.9138169Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.9138659Z ##[endgroup] 2024-06-01T03:36:46.9172227Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-01T03:36:46.9182601Z ##[error]Process completed with exit code 1. 2024-06-01T03:36:46.9215724Z ##[group]Run .github/scripts/parse_ref.py 2024-06-01T03:36:46.9216213Z .github/scripts/parse_ref.py 2024-06-01T03:36:46.9222866Z shell: /usr/bin/bash -e {0} 2024-06-01T03:36:46.9223347Z env: 2024-06-01T03:36:46.9223723Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.9224191Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.9224679Z ##[endgroup] 2024-06-01T03:36:46.9448866Z Prepare all required actions 2024-06-01T03:36:46.9487025Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-01T03:36:46.9487493Z with: 2024-06-01T03:36:46.9488359Z github-token: *** 2024-06-01T03:36:46.9488679Z env: 2024-06-01T03:36:46.9488955Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.9489418Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.9489916Z ##[endgroup] 2024-06-01T03:36:46.9504719Z ##[group]Run set -eux 2024-06-01T03:36:46.9505062Z set -eux 2024-06-01T03:36:46.9505669Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-01T03:36:46.9513416Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:46.9513924Z env: 2024-06-01T03:36:46.9514220Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:46.9514681Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:46.9515337Z GITHUB_TOKEN: *** 2024-06-01T03:36:46.9515648Z ##[endgroup] 2024-06-01T03:36:46.9534580Z + python3 .github/scripts/get_workflow_job_id.py 9326485603 i-0b128c63f91218fe4 2024-06-01T03:36:48.3377247Z setting job-id=25675761171 2024-06-01T03:36:48.3378639Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:36:48.3575864Z Prepare all required actions 2024-06-01T03:36:48.3576316Z Getting action download info 2024-06-01T03:36:48.4674160Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-01T03:36:48.4674617Z with: 2024-06-01T03:36:48.4675076Z github-token: *** 2024-06-01T03:36:48.4683392Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-01T03:36:48.4692186Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:36:48.4692956Z env: 2024-06-01T03:36:48.4693242Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:48.4693700Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:48.4694321Z ##[endgroup] 2024-06-01T03:36:48.4735662Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-01T03:36:48.4736221Z with: 2024-06-01T03:36:48.4736493Z shell: bash 2024-06-01T03:36:48.4736794Z timeout_minutes: 10 2024-06-01T03:36:48.4737122Z max_attempts: 5 2024-06-01T03:36:48.4737435Z retry_wait_seconds: 30 2024-06-01T03:36:48.4738584Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-01T03:36:48.4739781Z polling_interval_seconds: 1 2024-06-01T03:36:48.4740161Z warning_on_retry: true 2024-06-01T03:36:48.4740506Z continue_on_error: false 2024-06-01T03:36:48.4740843Z env: 2024-06-01T03:36:48.4741117Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:48.4741569Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:48.4742230Z GITHUB_TOKEN: *** 2024-06-01T03:36:48.4742557Z ##[endgroup] 2024-06-01T03:36:48.5230485Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-01T03:36:48.7319561Z Defaulting to user installation because normal site-packages is not writeable 2024-06-01T03:36:48.8538146Z Collecting requests==2.27.1 2024-06-01T03:36:48.8717851Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-06-01T03:36:49.0218327Z Collecting pyyaml==6.0.1 2024-06-01T03:36:49.0299898Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-06-01T03:36:49.3546920Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-06-01T03:36:49.3592213Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-06-01T03:36:49.4015045Z Collecting idna<4,>=2.5; python_version >= "3" 2024-06-01T03:36:49.4060829Z Downloading idna-3.7-py3-none-any.whl (66 kB) 2024-06-01T03:36:49.4669011Z Collecting certifi>=2017.4.17 2024-06-01T03:36:49.4710949Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-06-01T03:36:49.5824150Z Collecting urllib3<1.27,>=1.21.1 2024-06-01T03:36:49.5867554Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-06-01T03:36:49.6766008Z Installing collected packages: charset-normalizer, idna, certifi, urllib3, requests, pyyaml 2024-06-01T03:36:49.6991966Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-06-01T03:36:49.6993220Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-01T03:36:49.9299569Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.7 pyyaml-6.0.1 requests-2.27.1 urllib3-1.26.18 2024-06-01T03:36:50.5220594Z Command completed after 1 attempt(s). 2024-06-01T03:36:50.5260652Z ##[group]Run set -x 2024-06-01T03:36:50.5260999Z set -x 2024-06-01T03:36:50.5261303Z  2024-06-01T03:36:50.5261869Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-01T03:36:50.5262591Z # in runner workspace 2024-06-01T03:36:50.5263139Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-01T03:36:50.5271004Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:50.5271521Z env: 2024-06-01T03:36:50.5271794Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:50.5272255Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:50.5272754Z ##[endgroup] 2024-06-01T03:36:50.5293518Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-01T03:36:50.5490154Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-01T03:36:50.5490717Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-01T03:36:50.5491198Z echo "Job name: ${JOB_NAME}" 2024-06-01T03:36:50.5491614Z  2024-06-01T03:36:50.5492180Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-01T03:36:50.5493034Z # in runner workspace 2024-06-01T03:36:50.5493628Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-01T03:36:50.5494291Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-01T03:36:50.5494759Z  --job-name "${JOB_NAME}" \ 2024-06-01T03:36:50.5503303Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-01T03:36:50.5512087Z  --selected-test-configs "" \ 2024-06-01T03:36:50.5512609Z  --pr-number "${PR_NUMBER}" \ 2024-06-01T03:36:50.5513029Z  --tag "${TAG}" \ 2024-06-01T03:36:50.5513435Z  --event-name "${EVENT_NAME}" \ 2024-06-01T03:36:50.5513893Z  --schedule "${SCHEDULE}" \ 2024-06-01T03:36:50.5514337Z  --branch "${HEAD_BRANCH}" 2024-06-01T03:36:50.5521671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:50.5522177Z env: 2024-06-01T03:36:50.5522455Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:50.5523059Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:50.5523734Z GITHUB_TOKEN: *** 2024-06-01T03:36:50.5524435Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:36:50.5525216Z PR_NUMBER: 2024-06-01T03:36:50.5525526Z TAG: ciflow/inductor/127669 2024-06-01T03:36:50.5525892Z EVENT_NAME: push 2024-06-01T03:36:50.5526197Z SCHEDULE: 2024-06-01T03:36:50.5526481Z HEAD_BRANCH: 2024-06-01T03:36:50.5526776Z ##[endgroup] 2024-06-01T03:36:50.5548436Z Workflow: inductor 2024-06-01T03:36:50.5550732Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:36:50.7576425Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-01T03:36:50.8832901Z ##[group]Run echo "Filtered matrix:" 2024-06-01T03:36:50.8833374Z echo "Filtered matrix:" 2024-06-01T03:36:50.8842025Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-01T03:36:50.8850831Z  2024-06-01T03:36:50.8851103Z echo 2024-06-01T03:36:50.8851488Z echo "Is the current job unstable? False" 2024-06-01T03:36:50.8851955Z  2024-06-01T03:36:50.8852223Z echo 2024-06-01T03:36:50.8852586Z echo "Is keep-going label set? False" 2024-06-01T03:36:50.8853045Z  2024-06-01T03:36:50.8853317Z echo 2024-06-01T03:36:50.8853630Z echo "Renabled issues? " 2024-06-01T03:36:50.8861108Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:50.8861621Z env: 2024-06-01T03:36:50.8861901Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:50.8862354Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:50.8862852Z ##[endgroup] 2024-06-01T03:36:50.8882317Z Filtered matrix: 2024-06-01T03:36:50.8892579Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-01T03:36:50.8901257Z 2024-06-01T03:36:50.8901410Z Is the current job unstable? False 2024-06-01T03:36:50.8901706Z 2024-06-01T03:36:50.8901917Z Is keep-going label set? False 2024-06-01T03:36:50.8902189Z 2024-06-01T03:36:50.8902315Z Renabled issues? 2024-06-01T03:36:50.8944707Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-01T03:36:50.8945435Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-01T03:36:50.8953037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T03:36:50.8953542Z env: 2024-06-01T03:36:50.8953825Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:50.8954290Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:50.8954791Z JOB_TIMEOUT: 240 2024-06-01T03:36:50.8955109Z ##[endgroup] 2024-06-01T03:36:50.9034547Z ##[group]Run set -x 2024-06-01T03:36:50.9034953Z set -x 2024-06-01T03:36:50.9035257Z  2024-06-01T03:36:50.9035621Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-01T03:36:50.9036182Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-01T03:36:50.9036764Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-01T03:36:50.9037292Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-01T03:36:50.9037716Z else 2024-06-01T03:36:50.9038058Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-01T03:36:50.9038492Z fi 2024-06-01T03:36:50.9038769Z  2024-06-01T03:36:50.9039248Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-01T03:36:50.9040024Z # TODO: Stop building test binaries as part of the build phase 2024-06-01T03:36:50.9040724Z # Used for GPU_FLAG since that doesn't play nice 2024-06-01T03:36:50.9041327Z # shellcheck disable=SC2086,SC2090 2024-06-01T03:36:50.9041816Z container_name=$(docker run \ 2024-06-01T03:36:50.9042248Z  ${GPU_FLAG:-} \ 2024-06-01T03:36:50.9042766Z  -e BUILD_ENVIRONMENT \ 2024-06-01T03:36:50.9043177Z  -e PR_NUMBER \ 2024-06-01T03:36:50.9043539Z  -e GITHUB_ACTIONS \ 2024-06-01T03:36:50.9043942Z  -e GITHUB_REPOSITORY \ 2024-06-01T03:36:50.9044355Z  -e GITHUB_WORKFLOW \ 2024-06-01T03:36:50.9044750Z  -e GITHUB_JOB \ 2024-06-01T03:36:50.9045113Z  -e GITHUB_RUN_ID \ 2024-06-01T03:36:50.9045506Z  -e GITHUB_RUN_NUMBER \ 2024-06-01T03:36:50.9045920Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-01T03:36:50.9046318Z  -e JOB_ID \ 2024-06-01T03:36:50.9046663Z  -e JOB_NAME \ 2024-06-01T03:36:50.9047018Z  -e BASE_SHA \ 2024-06-01T03:36:50.9047367Z  -e BRANCH \ 2024-06-01T03:36:50.9047697Z  -e SHA1 \ 2024-06-01T03:36:50.9048054Z  -e AWS_DEFAULT_REGION \ 2024-06-01T03:36:50.9048466Z  -e IN_WHEEL_TEST \ 2024-06-01T03:36:50.9048850Z  -e SHARD_NUMBER \ 2024-06-01T03:36:50.9049219Z  -e TEST_CONFIG \ 2024-06-01T03:36:50.9049597Z  -e NUM_TEST_SHARDS \ 2024-06-01T03:36:50.9050000Z  -e REENABLED_ISSUES \ 2024-06-01T03:36:50.9050425Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-01T03:36:50.9050854Z  -e VERBOSE_TEST_LOGS \ 2024-06-01T03:36:50.9051267Z  -e NO_TEST_TIMEOUT \ 2024-06-01T03:36:50.9051655Z  -e NO_TD \ 2024-06-01T03:36:50.9051994Z  -e TD_DISTRIBUTED \ 2024-06-01T03:36:50.9052385Z  -e PR_LABELS \ 2024-06-01T03:36:50.9052803Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-01T03:36:50.9053273Z  -e SCCACHE_BUCKET \ 2024-06-01T03:36:50.9053670Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-01T03:36:50.9054091Z  -e XLA_CUDA \ 2024-06-01T03:36:50.9054503Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-01T03:36:50.9055151Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-01T03:36:50.9055682Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-01T03:36:50.9056210Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-01T03:36:50.9056692Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-01T03:36:50.9057123Z  -e DASHBOARD_TAG \ 2024-06-01T03:36:50.9057601Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-01T03:36:50.9058179Z  --security-opt seccomp=unconfined \ 2024-06-01T03:36:50.9058656Z  --cap-add=SYS_PTRACE \ 2024-06-01T03:36:50.9059054Z  --ipc=host \ 2024-06-01T03:36:50.9059412Z  --shm-size="${SHM_SIZE}" \ 2024-06-01T03:36:50.9059823Z  --tty \ 2024-06-01T03:36:50.9060143Z  --detach \ 2024-06-01T03:36:50.9060507Z  --name="${container_name}" \ 2024-06-01T03:36:50.9060927Z  --user jenkins \ 2024-06-01T03:36:50.9061498Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-01T03:36:50.9062137Z  -w /var/lib/jenkins/workspace \ 2024-06-01T03:36:50.9062588Z  "${DOCKER_IMAGE}" 2024-06-01T03:36:50.9062938Z ) 2024-06-01T03:36:50.9063359Z # Propagate download.pytorch.org IP to container 2024-06-01T03:36:50.9064327Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-01T03:36:50.9065340Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-01T03:36:50.9066286Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-01T03:36:50.9073687Z shell: /usr/bin/bash -e {0} 2024-06-01T03:36:50.9074042Z env: 2024-06-01T03:36:50.9074319Z GIT_DEFAULT_BRANCH: main 2024-06-01T03:36:50.9074773Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:36:50.9075402Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:36:50.9075926Z PR_NUMBER: 2024-06-01T03:36:50.9076250Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-01T03:36:50.9076667Z GITHUB_WORKFLOW: inductor 2024-06-01T03:36:50.9077024Z GITHUB_JOB: test 2024-06-01T03:36:50.9077344Z GITHUB_RUN_ID: 9326485603 2024-06-01T03:36:50.9077705Z GITHUB_RUN_NUMBER: 71862 2024-06-01T03:36:50.9078051Z GITHUB_RUN_ATTEMPT: 1 2024-06-01T03:36:50.9078388Z JOB_ID: 25675761171 2024-06-01T03:36:50.9079102Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:36:50.9079882Z BRANCH: 2024-06-01T03:36:50.9080219Z SHA1: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:36:50.9080760Z BASE_SHA: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:36:50.9081276Z TEST_CONFIG: dynamic_inductor_torchbench 2024-06-01T03:36:50.9081712Z SHARD_NUMBER: 2 2024-06-01T03:36:50.9082019Z NUM_TEST_SHARDS: 2 2024-06-01T03:36:50.9082392Z REENABLED_ISSUES: 2024-06-01T03:36:50.9082880Z CONTINUE_THROUGH_ERROR: False 2024-06-01T03:36:50.9083265Z VERBOSE_TEST_LOGS: False 2024-06-01T03:36:50.9083623Z NO_TEST_TIMEOUT: False 2024-06-01T03:36:50.9083958Z NO_TD: False 2024-06-01T03:36:50.9084261Z TD_DISTRIBUTED: False 2024-06-01T03:36:50.9084675Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-01T03:36:50.9085179Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-01T03:36:50.9085565Z SHM_SIZE: 2g 2024-06-01T03:36:50.9086700Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:36:50.9087916Z XLA_CUDA: 2024-06-01T03:36:50.9088409Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-01T03:36:50.9089044Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-01T03:36:50.9089484Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-01T03:36:50.9089893Z DASHBOARD_TAG: 2024-06-01T03:36:50.9090410Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-01T03:36:50.9090872Z ##[endgroup] 2024-06-01T03:36:50.9110472Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-01T03:36:50.9111279Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-01T03:36:50.9111855Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-01T03:36:50.9118170Z +++ nproc --ignore=2 2024-06-01T03:36:50.9144588Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9326485603 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T03:37:02.3351640Z + container_name=115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T03:37:02.3352997Z + grep download.pytorch.org /etc/hosts 2024-06-01T03:37:02.3354290Z + docker exec -i 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-01T03:37:02.3989307Z + echo DOCKER_CONTAINER_ID=115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T03:37:02.3992105Z ++ echo dist/torch-2.4.0a0+gitde352ff-cp310-cp310-linux_x86_64.whl 2024-06-01T03:37:02.3994381Z + docker exec -t 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 sh -c 'pip install dist/torch-2.4.0a0+gitde352ff-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-01T03:37:02.7919192Z Processing ./dist/torch-2.4.0a0+gitde352ff-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitde352ff) 2024-06-01T03:37:03.1052570Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (3.13.1) 2024-06-01T03:37:03.1056012Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (4.12.0) 2024-06-01T03:37:03.1058254Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (1.12.1) 2024-06-01T03:37:03.1061722Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (2.8.8) 2024-06-01T03:37:03.1065019Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (3.1.4) 2024-06-01T03:37:03.1068287Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (2024.2.0) 2024-06-01T03:37:03.1097294Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (3.3.0) 2024-06-01T03:37:03.1156782Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (1.21.2) 2024-06-01T03:37:03.1595220Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (2.1.5) 2024-06-01T03:37:03.1756882Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitde352ff->torch==2.4.0a0+gitde352ff) (1.3.0) 2024-06-01T03:37:04.5203884Z Installing collected packages: torch 2024-06-01T03:37:13.1813963Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-01T03:37:13.1815358Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-01T03:37:13.1816676Z Successfully installed torch-2.4.0a0+gitde352ff 2024-06-01T03:37:13.2746607Z ++ dirname .ci/pytorch/test.sh 2024-06-01T03:37:13.2749872Z + source .ci/pytorch/common.sh 2024-06-01T03:37:13.2752523Z +++ dirname .ci/pytorch/common.sh 2024-06-01T03:37:13.2759660Z ++ source .ci/pytorch/common_utils.sh 2024-06-01T03:37:13.2760638Z +++ declare -f -t trap_add 2024-06-01T03:37:13.2766941Z ++ set -ex 2024-06-01T03:37:13.2767523Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-01T03:37:13.2768127Z ++ BUILD_TEST_LIBTORCH=0 2024-06-01T03:37:13.2768638Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-01T03:37:13.2770638Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-01T03:37:13.2778464Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-01T03:37:13.2778974Z + trap_add cleanup_workspace EXIT 2024-06-01T03:37:13.2779435Z + trap_add_cmd=cleanup_workspace 2024-06-01T03:37:13.2779902Z + shift 2024-06-01T03:37:13.2780771Z + for trap_add_name in "$@" 2024-06-01T03:37:13.2783943Z +++ trap -p EXIT 2024-06-01T03:37:13.2785293Z ++ eval 'extract_trap_cmd ' 2024-06-01T03:37:13.2785807Z +++ extract_trap_cmd 2024-06-01T03:37:13.2786274Z +++ printf '%s\n' '' 2024-06-01T03:37:13.2786666Z ++ printf '%s\n' cleanup_workspace 2024-06-01T03:37:13.2787150Z + trap -- ' 2024-06-01T03:37:13.2787542Z cleanup_workspace' EXIT 2024-06-01T03:37:13.2788012Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-01T03:37:13.7659066Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-01T03:37:13.7672101Z + echo 'Environment variables:' 2024-06-01T03:37:13.7672605Z Environment variables: 2024-06-01T03:37:13.7672950Z + env 2024-06-01T03:37:13.7676916Z INSTALLED_DB=yes 2024-06-01T03:37:13.7677488Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-06-01T03:37:13.7677918Z NVIDIA_VISIBLE_DEVICES=all 2024-06-01T03:37:13.7678338Z NV_NVML_DEV_VERSION=12.4.99-1 2024-06-01T03:37:13.7679126Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T03:37:13.7679831Z CONTINUE_THROUGH_ERROR=False 2024-06-01T03:37:13.7680369Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-06-01T03:37:13.7680907Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-06-01T03:37:13.7681632Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:37:13.7682266Z HOSTNAME=115e64e466b1 2024-06-01T03:37:13.7683407Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7684373Z GITHUB_ACTION=__self 2024-06-01T03:37:13.7684856Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-01T03:37:13.7690711Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-06-01T03:37:13.7697059Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-06-01T03:37:13.7697571Z NV_NVTX_VERSION=12.4.99-1 2024-06-01T03:37:13.7697926Z GITHUB_RUN_NUMBER=71862 2024-06-01T03:37:13.7698295Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-01T03:37:13.7698738Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-01T03:37:13.7699208Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-01T03:37:13.7699671Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-06-01T03:37:13.7700107Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-06-01T03:37:13.7700538Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-06-01T03:37:13.7701083Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-01T03:37:13.7701589Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-01T03:37:13.7702050Z GITHUB_REF_TYPE=tag 2024-06-01T03:37:13.7702377Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-01T03:37:13.7702758Z NCCL_VERSION=2.20.5-1 2024-06-01T03:37:13.7703149Z BASE_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7703604Z XLA_CUDA= 2024-06-01T03:37:13.7704096Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-01T03:37:13.7705694Z *** 2024-06-01T03:37:13.7705988Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-01T03:37:13.7706388Z GITHUB_REPOSITORY_ID=65600975 2024-06-01T03:37:13.7706757Z GITHUB_ACTIONS=true 2024-06-01T03:37:13.7707097Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:37:13.7707593Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-06-01T03:37:13.7708133Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-06-01T03:37:13.7708613Z SHA1=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7709132Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-01T03:37:13.7709631Z GITHUB_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7710815Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/127669 2024-06-01T03:37:13.7711579Z UCC_HOME=/usr 2024-06-01T03:37:13.7711935Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-06-01T03:37:13.7712356Z VERBOSE_TEST_LOGS=False 2024-06-01T03:37:13.7712711Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-01T03:37:13.7713199Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-06-01T03:37:13.7713729Z GITHUB_REF=refs/tags/ciflow/inductor/127669 2024-06-01T03:37:13.7714216Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-06-01T03:37:13.7714610Z SHARD_NUMBER=2 2024-06-01T03:37:13.7714924Z GITHUB_REF_PROTECTED=false 2024-06-01T03:37:13.7715283Z HOME=/var/lib/jenkins 2024-06-01T03:37:13.7715651Z GITHUB_API_URL=https://api.github.com 2024-06-01T03:37:13.7716096Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-01T03:37:13.7716581Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-01T03:37:13.7717106Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-01T03:37:13.7717506Z CUDA_VERSION=12.4.0 2024-06-01T03:37:13.7717932Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-06-01T03:37:13.7718382Z NUM_TEST_SHARDS=2 2024-06-01T03:37:13.7718682Z UCX_HOME=/usr 2024-06-01T03:37:13.7719201Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-06-01T03:37:13.7720300Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7721594Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:37:13.7723012Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7724155Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-01T03:37:13.7724822Z GITHUB_EVENT_NAME=push 2024-06-01T03:37:13.7725156Z DASHBOARD_TAG= 2024-06-01T03:37:13.7725596Z GITHUB_RUN_ID=9326485603 2024-06-01T03:37:13.7726061Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-06-01T03:37:13.7726609Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-06-01T03:37:13.7727640Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7728588Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-01T03:37:13.7729003Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-06-01T03:37:13.7729384Z PR_NUMBER= 2024-06-01T03:37:13.7729671Z GITHUB_RUN_ATTEMPT=1 2024-06-01T03:37:13.7730008Z ANACONDA_PYTHON_VERSION=3.10 2024-06-01T03:37:13.7730464Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-01T03:37:13.7730934Z TERM=xterm 2024-06-01T03:37:13.7731262Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-06-01T03:37:13.7731684Z INSTALLED_VISION=yes 2024-06-01T03:37:13.7731998Z BRANCH= 2024-06-01T03:37:13.7732287Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-01T03:37:13.7732689Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-01T03:37:13.7733219Z CUDA_PATH=/usr/local/cuda 2024-06-01T03:37:13.7734001Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-01T03:37:13.7734788Z GITHUB_SERVER_URL=https://github.com 2024-06-01T03:37:13.7735282Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-01T03:37:13.7735764Z REENABLED_ISSUES= 2024-06-01T03:37:13.7736064Z SHLVL=1 2024-06-01T03:37:13.7736323Z MAX_JOBS=14 2024-06-01T03:37:13.7736637Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-06-01T03:37:13.7736995Z NVARCH=x86_64 2024-06-01T03:37:13.7737296Z GITHUB_ACTOR_ID=54816060 2024-06-01T03:37:13.7737762Z GITHUB_WORKFLOW_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7738319Z GITHUB_REF_NAME=ciflow/inductor/127669 2024-06-01T03:37:13.7738818Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-06-01T03:37:13.7739499Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-01T03:37:13.7740088Z GITHUB_JOB=test 2024-06-01T03:37:13.7740496Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-06-01T03:37:13.7741075Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-01T03:37:13.7741594Z NO_TEST_TIMEOUT=False 2024-06-01T03:37:13.7741927Z TD_DISTRIBUTED=False 2024-06-01T03:37:13.7742307Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-06-01T03:37:13.7742758Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-01T03:37:13.7743177Z NV_NVPROF_VERSION=12.4.99-1 2024-06-01T03:37:13.7743548Z GITHUB_RETENTION_DAYS=90 2024-06-01T03:37:13.7743901Z OPENSSL_DIR=/opt/openssl 2024-06-01T03:37:13.7744253Z GITHUB_ACTION_REPOSITORY= 2024-06-01T03:37:13.7745326Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-01T03:37:13.7746440Z GITHUB_BASE_REF= 2024-06-01T03:37:13.7746768Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-01T03:37:13.7747149Z CI=true 2024-06-01T03:37:13.7747456Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-06-01T03:37:13.7747887Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-01T03:37:13.7748273Z JOB_ID=25675761171 2024-06-01T03:37:13.7748591Z INSTALLED_PROTOBUF=yes 2024-06-01T03:37:13.7748926Z GITHUB_HEAD_REF= 2024-06-01T03:37:13.7749233Z GITHUB_ACTION_REF= 2024-06-01T03:37:13.7749667Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-01T03:37:13.7750490Z GITHUB_WORKFLOW=inductor 2024-06-01T03:37:13.7750880Z DEBIAN_FRONTEND=noninteractive 2024-06-01T03:37:13.7751818Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7752673Z NO_TD=False 2024-06-01T03:37:13.7752974Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-01T03:37:13.7753346Z _=/usr/bin/env 2024-06-01T03:37:13.7753820Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-01T03:37:13.7823571Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-01T03:37:13.7824802Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-01T03:37:13.7825970Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-01T03:37:13.7826878Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-01T03:37:13.7827509Z + BUILD_DIR=build 2024-06-01T03:37:13.7827840Z + BUILD_RENAMED_DIR=build_renamed 2024-06-01T03:37:13.7828245Z + BUILD_BIN_DIR=build/bin 2024-06-01T03:37:13.7828604Z + SHARD_NUMBER=2 2024-06-01T03:37:13.7828913Z + NUM_TEST_SHARDS=2 2024-06-01T03:37:13.7829231Z + export VALGRIND=ON 2024-06-01T03:37:13.7829558Z + VALGRIND=ON 2024-06-01T03:37:13.7830307Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-01T03:37:13.7830821Z + [[ 0 == \1 ]] 2024-06-01T03:37:13.7831130Z + [[ False == \1 ]] 2024-06-01T03:37:13.7831621Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-01T03:37:13.7832170Z ++ realpath build/custom_test_artifacts 2024-06-01T03:37:13.7835079Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-01T03:37:13.7836011Z + [[ -n '' ]] 2024-06-01T03:37:13.7836385Z + echo 'Environment variables' 2024-06-01T03:37:13.7836796Z Environment variables 2024-06-01T03:37:13.7837171Z + env 2024-06-01T03:37:13.7840215Z INSTALLED_DB=yes 2024-06-01T03:37:13.7841454Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-06-01T03:37:13.7842150Z NVIDIA_VISIBLE_DEVICES=all 2024-06-01T03:37:13.7842748Z NV_NVML_DEV_VERSION=12.4.99-1 2024-06-01T03:37:13.7843447Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T03:37:13.7844180Z CONTINUE_THROUGH_ERROR=False 2024-06-01T03:37:13.7844773Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-06-01T03:37:13.7845465Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-06-01T03:37:13.7846404Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T03:37:13.7846940Z HOSTNAME=115e64e466b1 2024-06-01T03:37:13.7847818Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7848683Z GITHUB_ACTION=__self 2024-06-01T03:37:13.7849050Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-01T03:37:13.7854838Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-06-01T03:37:13.7862745Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-06-01T03:37:13.7863267Z NV_NVTX_VERSION=12.4.99-1 2024-06-01T03:37:13.7863619Z GITHUB_RUN_NUMBER=71862 2024-06-01T03:37:13.7863995Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-01T03:37:13.7864448Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-01T03:37:13.7864927Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-01T03:37:13.7865385Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-06-01T03:37:13.7865831Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-06-01T03:37:13.7866265Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-06-01T03:37:13.7866701Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-01T03:37:13.7867312Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-01T03:37:13.7867782Z GITHUB_REF_TYPE=tag 2024-06-01T03:37:13.7868365Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-01T03:37:13.7868754Z NCCL_VERSION=2.20.5-1 2024-06-01T03:37:13.7869151Z BASE_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7869612Z XLA_CUDA= 2024-06-01T03:37:13.7870282Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-01T03:37:13.7870695Z *** 2024-06-01T03:37:13.7870981Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-01T03:37:13.7871382Z GITHUB_REPOSITORY_ID=65600975 2024-06-01T03:37:13.7871754Z GITHUB_ACTIONS=true 2024-06-01T03:37:13.7872091Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T03:37:13.7872583Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-06-01T03:37:13.7873125Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-06-01T03:37:13.7873606Z SHA1=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7874131Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-01T03:37:13.7874624Z GITHUB_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7875528Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/127669 2024-06-01T03:37:13.7876447Z UCC_HOME=/usr 2024-06-01T03:37:13.7876802Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-06-01T03:37:13.7877212Z VERBOSE_TEST_LOGS=False 2024-06-01T03:37:13.7877559Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-01T03:37:13.7878019Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-06-01T03:37:13.7878532Z GITHUB_REF=refs/tags/ciflow/inductor/127669 2024-06-01T03:37:13.7878999Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-06-01T03:37:13.7879378Z SHARD_NUMBER=2 2024-06-01T03:37:13.7879689Z GITHUB_REF_PROTECTED=false 2024-06-01T03:37:13.7880038Z HOME=/var/lib/jenkins 2024-06-01T03:37:13.7880401Z GITHUB_API_URL=https://api.github.com 2024-06-01T03:37:13.7880846Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-01T03:37:13.7881329Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-01T03:37:13.7881815Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-01T03:37:13.7882192Z CUDA_VERSION=12.4.0 2024-06-01T03:37:13.7882722Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-06-01T03:37:13.7883188Z NUM_TEST_SHARDS=2 2024-06-01T03:37:13.7883484Z UCX_HOME=/usr 2024-06-01T03:37:13.7884007Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-06-01T03:37:13.7885121Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7886439Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-01T03:37:13.7887730Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7888893Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-01T03:37:13.7889574Z GITHUB_EVENT_NAME=push 2024-06-01T03:37:13.7889909Z DASHBOARD_TAG= 2024-06-01T03:37:13.7890205Z GITHUB_RUN_ID=9326485603 2024-06-01T03:37:13.7890658Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-06-01T03:37:13.7891202Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-06-01T03:37:13.7892245Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7893191Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-01T03:37:13.7893601Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-06-01T03:37:13.7893987Z PR_NUMBER= 2024-06-01T03:37:13.7894268Z GITHUB_RUN_ATTEMPT=1 2024-06-01T03:37:13.7894586Z VALGRIND=ON 2024-06-01T03:37:13.7894890Z ANACONDA_PYTHON_VERSION=3.10 2024-06-01T03:37:13.7895347Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-01T03:37:13.7895813Z TERM=xterm 2024-06-01T03:37:13.7896152Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-06-01T03:37:13.7896576Z INSTALLED_VISION=yes 2024-06-01T03:37:13.7896894Z BRANCH= 2024-06-01T03:37:13.7897202Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-01T03:37:13.7897635Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-01T03:37:13.7898074Z CUDA_PATH=/usr/local/cuda 2024-06-01T03:37:13.7898840Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-01T03:37:13.7899731Z GITHUB_SERVER_URL=https://github.com 2024-06-01T03:37:13.7900232Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-01T03:37:13.7900712Z REENABLED_ISSUES= 2024-06-01T03:37:13.7901007Z SHLVL=1 2024-06-01T03:37:13.7901270Z MAX_JOBS=14 2024-06-01T03:37:13.7901585Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-06-01T03:37:13.7901944Z NVARCH=x86_64 2024-06-01T03:37:13.7902237Z GITHUB_ACTOR_ID=54816060 2024-06-01T03:37:13.7902710Z GITHUB_WORKFLOW_SHA=de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T03:37:13.7903267Z GITHUB_REF_NAME=ciflow/inductor/127669 2024-06-01T03:37:13.7903759Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-06-01T03:37:13.7904440Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-01T03:37:13.7905031Z GITHUB_JOB=test 2024-06-01T03:37:13.7905429Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-06-01T03:37:13.7905994Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-01T03:37:13.7906581Z NO_TEST_TIMEOUT=False 2024-06-01T03:37:13.7906917Z TD_DISTRIBUTED=False 2024-06-01T03:37:13.7907346Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-06-01T03:37:13.7907789Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-01T03:37:13.7908215Z NV_NVPROF_VERSION=12.4.99-1 2024-06-01T03:37:13.7908584Z GITHUB_RETENTION_DAYS=90 2024-06-01T03:37:13.7908939Z OPENSSL_DIR=/opt/openssl 2024-06-01T03:37:13.7909287Z GITHUB_ACTION_REPOSITORY= 2024-06-01T03:37:13.7910503Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-01T03:37:13.7911621Z GITHUB_BASE_REF= 2024-06-01T03:37:13.7911945Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-01T03:37:13.7912326Z CI=true 2024-06-01T03:37:13.7912639Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-06-01T03:37:13.7913067Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-01T03:37:13.7913442Z JOB_ID=25675761171 2024-06-01T03:37:13.7913758Z INSTALLED_PROTOBUF=yes 2024-06-01T03:37:13.7914098Z GITHUB_HEAD_REF= 2024-06-01T03:37:13.7914409Z GITHUB_ACTION_REF= 2024-06-01T03:37:13.7914840Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-01T03:37:13.7915327Z GITHUB_WORKFLOW=inductor 2024-06-01T03:37:13.7915693Z DEBIAN_FRONTEND=noninteractive 2024-06-01T03:37:13.7916624Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f6c79e9c-e7c2-43d3-b901-77b1827011b9 2024-06-01T03:37:13.7917482Z NO_TD=False 2024-06-01T03:37:13.7917785Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-01T03:37:13.7918152Z _=/usr/bin/env 2024-06-01T03:37:13.7918480Z + echo 'Testing pytorch' 2024-06-01T03:37:13.7918824Z Testing pytorch 2024-06-01T03:37:13.7919143Z + export LANG=C.UTF-8 2024-06-01T03:37:13.7919477Z + LANG=C.UTF-8 2024-06-01T03:37:13.7919761Z + PR_NUMBER= 2024-06-01T03:37:13.7920139Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-06-01T03:37:13.7920738Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-01T03:37:13.7921303Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-06-01T03:37:13.7921978Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-01T03:37:13.7922776Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-01T03:37:13.7923337Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-01T03:37:13.7923817Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-01T03:37:13.7924300Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-06-01T03:37:13.7924909Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-01T03:37:13.7925549Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-01T03:37:13.7926195Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-01T03:37:13.7926764Z + pip_install --user ninja==1.10.2 2024-06-01T03:37:13.7927307Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-01T03:37:14.1472721Z Collecting ninja==1.10.2 2024-06-01T03:37:14.1619375Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-01T03:37:14.2008677Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-01T03:37:15.5258516Z Installing collected packages: ninja 2024-06-01T03:37:15.5326191Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-01T03:37:15.5327694Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-01T03:37:15.5367196Z Successfully installed ninja-1.10.2 2024-06-01T03:37:15.6336747Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-01T03:37:15.6338955Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-01T03:37:15.6340840Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-01T03:37:15.6341385Z + install_tlparse 2024-06-01T03:37:15.6341778Z + pip_install --user tlparse==0.3.7 2024-06-01T03:37:15.6342337Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-01T03:37:15.9861910Z Collecting tlparse==0.3.7 2024-06-01T03:37:16.0004552Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-01T03:37:16.0110769Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-01T03:37:17.3470594Z Installing collected packages: tlparse 2024-06-01T03:37:17.3837016Z Successfully installed tlparse-0.3.7 2024-06-01T03:37:17.4731910Z ++ python -m site --user-base 2024-06-01T03:37:17.4883647Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-01T03:37:17.4885641Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-06-01T03:37:17.4886568Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-01T03:37:17.4887263Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-01T03:37:17.4888487Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-01T03:37:17.4889696Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-01T03:37:17.4890432Z + cd test 2024-06-01T03:37:17.4890968Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-01T03:37:18.9400636Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-01T03:37:18.9401479Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-01T03:37:18.9402529Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-01T03:37:18.9403119Z + pushd test 2024-06-01T03:37:18.9403435Z ~/workspace/test ~/workspace 2024-06-01T03:37:18.9404002Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-01T03:37:20.3844164Z + CUDA_VERSION=12.4 2024-06-01T03:37:20.3844584Z + '[' 12.4 == 12.4 ']' 2024-06-01T03:37:20.3844927Z + ISCUDA124=cu124 2024-06-01T03:37:20.3845225Z + popd 2024-06-01T03:37:20.3845495Z ~/workspace 2024-06-01T03:37:20.3849187Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-01T03:37:20.3849805Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-06-01T03:37:20.3850550Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-06-01T03:37:20.3851293Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-01T03:37:20.3851890Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-06-01T03:37:20.3852408Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-06-01T03:37:20.3852943Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-01T03:37:20.3853431Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-06-01T03:37:20.3854099Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-01T03:37:20.3854759Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-01T03:37:20.3855505Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-01T03:37:20.3876539Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-01T03:37:20.3877244Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-01T03:37:20.3878643Z + cd test 2024-06-01T03:37:20.3879204Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-01T03:37:21.7152120Z PyTorch built with: 2024-06-01T03:37:21.7152640Z - GCC 9.4 2024-06-01T03:37:21.7153075Z - C++ Version: 201703 2024-06-01T03:37:21.7154074Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-01T03:37:21.7155183Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-01T03:37:21.7155863Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-01T03:37:21.7156390Z - LAPACK is enabled (usually provided by MKL) 2024-06-01T03:37:21.7156875Z - NNPACK is enabled 2024-06-01T03:37:21.7157448Z - CPU capability usage: AVX2 2024-06-01T03:37:21.7157871Z - CUDA Runtime 12.4 2024-06-01T03:37:21.7158413Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-01T03:37:21.7159025Z - CuDNN 8.9.7 (built against CUDA 12.2) 2024-06-01T03:37:21.7159480Z - Magma 2.6.1 2024-06-01T03:37:21.7167110Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=8.9.7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-01T03:37:21.7173922Z 2024-06-01T03:37:21.9922030Z + cd test 2024-06-01T03:37:21.9922896Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-01T03:37:23.1769508Z ATen/Parallel: 2024-06-01T03:37:23.1770080Z at::get_num_threads() : 8 2024-06-01T03:37:23.1770607Z at::get_num_interop_threads() : 16 2024-06-01T03:37:23.1771190Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-01T03:37:23.1771753Z omp_get_max_threads() : 8 2024-06-01T03:37:23.1772849Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-01T03:37:23.1773684Z mkl_get_max_threads() : 8 2024-06-01T03:37:23.1774325Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-01T03:37:23.1774986Z std::thread::hardware_concurrency() : 16 2024-06-01T03:37:23.1775435Z Environment variables: 2024-06-01T03:37:23.1775789Z OMP_NUM_THREADS : [not set] 2024-06-01T03:37:23.1776174Z MKL_NUM_THREADS : [not set] 2024-06-01T03:37:23.1776574Z ATen parallel backend: OpenMP 2024-06-01T03:37:23.1776836Z 2024-06-01T03:37:23.4190528Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-01T03:37:23.4191389Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-06-01T03:37:23.4192085Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-06-01T03:37:23.4192628Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-06-01T03:37:23.4193638Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-01T03:37:23.4194550Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-01T03:37:23.4195348Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-06-01T03:37:23.4196055Z + [[ dynamic_inductor_torchbench == deploy ]] 2024-06-01T03:37:23.4196823Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-06-01T03:37:23.4197860Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-06-01T03:37:23.4198730Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-06-01T03:37:23.4199450Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-06-01T03:37:23.4200208Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-06-01T03:37:23.4200996Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-01T03:37:23.4201709Z + install_torchaudio cuda 2024-06-01T03:37:23.4202101Z + local commit 2024-06-01T03:37:23.4202523Z ++ get_pinned_commit audio 2024-06-01T03:37:23.4202915Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-01T03:37:23.4206904Z + commit=1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:23.4207787Z + [[ cuda == \c\u\d\a ]] 2024-06-01T03:37:23.4208290Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-01T03:37:23.4209308Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:23.4210980Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:23.7332228Z Collecting git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:23.7336783Z Cloning https://github.com/pytorch/audio.git (to revision 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0) to /tmp/pip-req-build-j98w4f4k 2024-06-01T03:37:23.7352409Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-j98w4f4k 2024-06-01T03:37:24.4421510Z Running command git rev-parse -q --verify 'sha^1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0' 2024-06-01T03:37:24.4440842Z Running command git fetch -q https://github.com/pytorch/audio.git 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:24.8340839Z Resolved https://github.com/pytorch/audio.git to commit 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-06-01T03:37:24.8342026Z Running command git submodule update --init --recursive -q 2024-06-01T03:37:26.9443764Z Preparing metadata (setup.py) ... [?25l- done 2024-06-01T03:37:26.9473704Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.2.0a0+1980f8a) (2.4.0a0+gitde352ff) 2024-06-01T03:37:26.9535399Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.13.1) 2024-06-01T03:37:26.9540539Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (4.12.0) 2024-06-01T03:37:26.9542896Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (1.12.1) 2024-06-01T03:37:26.9545811Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2.8.8) 2024-06-01T03:37:26.9548629Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.1.4) 2024-06-01T03:37:26.9551764Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2024.2.0) 2024-06-01T03:37:26.9981550Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.2.0a0+1980f8a) (2.1.5) 2024-06-01T03:37:27.0135779Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.2.0a0+1980f8a) (1.3.0) 2024-06-01T03:37:27.0206581Z Building wheels for collected packages: torchaudio 2024-06-01T03:39:34.1288484Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-01T03:39:34.1316695Z [?25h Created wheel for torchaudio: filename=torchaudio-2.2.0a0+1980f8a-cp310-cp310-linux_x86_64.whl size=2311884 sha256=66d383548a57976d23b2b21bd594fafe7d8940a26fb9c2e2b34b9fcba0ecba7a 2024-06-01T03:39:34.1318388Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3c/69/04/2f8953929b90f7f98a00671a3b24cfeb70fc70e57328823e5c 2024-06-01T03:39:34.1344965Z Successfully built torchaudio 2024-06-01T03:39:35.3556288Z Installing collected packages: torchaudio 2024-06-01T03:39:35.5465844Z Successfully installed torchaudio-2.2.0a0+1980f8a 2024-06-01T03:39:35.6798589Z + install_torchtext 2024-06-01T03:39:35.6799047Z + local data_commit 2024-06-01T03:39:35.6803232Z + local text_commit 2024-06-01T03:39:35.6803586Z ++ get_pinned_commit data 2024-06-01T03:39:35.6803980Z ++ cat .github/ci_commit_pins/data.txt 2024-06-01T03:39:35.6812235Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:35.6813739Z ++ get_pinned_commit text 2024-06-01T03:39:35.6814124Z ++ cat .github/ci_commit_pins/text.txt 2024-06-01T03:39:35.6821418Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:39:35.6822539Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:35.6824033Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:35.9953489Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:35.9956839Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-nw47vq15 2024-06-01T03:39:35.9974861Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-nw47vq15 2024-06-01T03:39:36.4553368Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-01T03:39:36.4567642Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:36.6937288Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:36.8643905Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-01T03:39:36.8645153Z Running command git submodule update --init --recursive -q 2024-06-01T03:40:44.6935902Z Preparing metadata (setup.py) ... [?25l- done 2024-06-01T03:40:44.6987524Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.18) 2024-06-01T03:40:44.6990756Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-01T03:40:44.6996625Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.4.0a0+gitde352ff) 2024-06-01T03:40:44.7057848Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-01T03:40:44.7063364Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.0) 2024-06-01T03:40:44.7066630Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-01T03:40:44.7070285Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-01T03:40:44.7073504Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-01T03:40:44.7076715Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-01T03:40:44.7265727Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-01T03:40:44.7271109Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-01T03:40:44.7277436Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.2.2) 2024-06-01T03:40:44.7742585Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-01T03:40:44.7918054Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-01T03:40:44.7997288Z Building wheels for collected packages: torchdata 2024-06-01T03:40:46.5218844Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-01T03:40:46.5226184Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=aba07d5767be90358af797c3539ea903701fad35ddde92041ea373fc80c28f1a 2024-06-01T03:40:46.5229005Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-01T03:40:46.5253707Z Successfully built torchdata 2024-06-01T03:40:47.7359539Z Installing collected packages: torchdata 2024-06-01T03:40:47.8208872Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-01T03:40:50.2693296Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:50.2694827Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:50.5806811Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:50.5812029Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-rqrhbmhr 2024-06-01T03:40:50.5826523Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-rqrhbmhr 2024-06-01T03:40:51.3766479Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-01T03:40:51.3785141Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:51.8700122Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:52.0473685Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-01T03:40:52.0474772Z Running command git submodule update --init --recursive -q 2024-06-01T03:40:59.0405705Z Preparing metadata (setup.py) ... [?25l- done 2024-06-01T03:40:59.0456168Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-01T03:40:59.0458434Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-01T03:40:59.0461606Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.4.0a0+gitde352ff) 2024-06-01T03:40:59.0464212Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-01T03:40:59.0467140Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-01T03:40:59.0530303Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-01T03:40:59.0534981Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-01T03:40:59.0541332Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.18) 2024-06-01T03:40:59.0548022Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.2.2) 2024-06-01T03:40:59.0604566Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-01T03:40:59.0610962Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.0) 2024-06-01T03:40:59.0613961Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-01T03:40:59.0616654Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-01T03:40:59.0619427Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-01T03:40:59.0621995Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-01T03:40:59.1302784Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-01T03:40:59.1469250Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-01T03:40:59.1550828Z Building wheels for collected packages: torchtext 2024-06-01T03:41:35.2308749Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-01T03:41:35.2336076Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045369 sha256=487de558d2f8edb80ff1893124f157690a56992d5a2ef1dee8c9315d76b1c093 2024-06-01T03:41:35.2338089Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-01T03:41:35.2365035Z Successfully built torchtext 2024-06-01T03:41:36.4312136Z Installing collected packages: torchtext 2024-06-01T03:41:36.5287283Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-01T03:41:36.6504238Z + install_torchvision 2024-06-01T03:41:36.6506754Z + local orig_preload 2024-06-01T03:41:36.6507691Z + local commit 2024-06-01T03:41:36.6508571Z ++ get_pinned_commit vision 2024-06-01T03:41:36.6509573Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-01T03:41:36.6519931Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:36.6520408Z + orig_preload= 2024-06-01T03:41:36.6521001Z + '[' -n '' ']' 2024-06-01T03:41:36.6521855Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:36.6523441Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:36.9637103Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:36.9641037Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-ok3ui6c5 2024-06-01T03:41:36.9657821Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-ok3ui6c5 2024-06-01T03:41:38.3676726Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-01T03:41:38.3694357Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:39.5778023Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:39.8092877Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-01T03:41:42.0907494Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-01T03:41:42.0954904Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-01T03:41:42.0957853Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitde352ff) 2024-06-01T03:41:42.0963204Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-01T03:41:42.1165676Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-01T03:41:42.1171697Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.0) 2024-06-01T03:41:42.1174187Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-01T03:41:42.1177199Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-01T03:41:42.1179518Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-01T03:41:42.1182634Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-01T03:41:42.1610211Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-01T03:41:42.1763569Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-01T03:41:42.1839879Z Building wheels for collected packages: torchvision 2024-06-01T03:42:52.5677797Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-01T03:42:52.5704906Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2025747 sha256=502d6b2fb330df16fc0214518c8761f30ae92813245daaf34813fb22bcecd6bc 2024-06-01T03:42:52.5708233Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-01T03:42:52.5740324Z Successfully built torchvision 2024-06-01T03:42:53.7806042Z Installing collected packages: torchvision 2024-06-01T03:42:54.1753913Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-01T03:42:54.2844334Z + '[' -n '' ']' 2024-06-01T03:42:54.2844898Z + id=1 2024-06-01T03:42:54.2845527Z + pip_install opencv-python==4.8.0.74 2024-06-01T03:42:54.2849396Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-01T03:42:54.7136795Z Collecting opencv-python==4.8.0.74 2024-06-01T03:42:54.7285932Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-01T03:42:54.7480700Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-01T03:42:54.7557041Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-01T03:42:56.6077636Z Installing collected packages: opencv-python 2024-06-01T03:42:57.4260315Z Successfully installed opencv-python-4.8.0.74 2024-06-01T03:42:57.5213101Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-01T03:42:57.5215306Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-01T03:42:57.5217296Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-01T03:42:57.5218551Z + checkout_install_torchbench 2024-06-01T03:42:57.5219281Z + local commit 2024-06-01T03:42:57.5219903Z ++ get_pinned_commit torchbench 2024-06-01T03:42:57.5220768Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-01T03:42:57.5227167Z + commit=d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-06-01T03:42:57.5227785Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-01T03:42:57.5236754Z Cloning into 'torchbench'... 2024-06-01T03:42:57.6628502Z remote: Enumerating objects: 29396, done. 2024-06-01T03:42:57.6629539Z remote: Counting objects: 0% (1/4376) 2024-06-01T03:42:57.6630374Z remote: Counting objects: 1% (44/4376) 2024-06-01T03:42:57.6631061Z remote: Counting objects: 2% (88/4376) 2024-06-01T03:42:57.6631603Z remote: Counting objects: 3% (132/4376) 2024-06-01T03:42:57.6632135Z remote: Counting objects: 4% (176/4376) 2024-06-01T03:42:57.6632671Z remote: Counting objects: 5% (219/4376) 2024-06-01T03:42:57.6633224Z remote: Counting objects: 6% (263/4376) 2024-06-01T03:42:57.6633891Z remote: Counting objects: 7% (307/4376) 2024-06-01T03:42:57.6634413Z remote: Counting objects: 8% (351/4376) 2024-06-01T03:42:57.6634938Z remote: Counting objects: 9% (394/4376) 2024-06-01T03:42:57.6635529Z remote: Counting objects: 10% (438/4376) 2024-06-01T03:42:57.6636257Z remote: Counting objects: 11% (482/4376) 2024-06-01T03:42:57.6636992Z remote: Counting objects: 12% (526/4376) 2024-06-01T03:42:57.6637713Z remote: Counting objects: 13% (569/4376) 2024-06-01T03:42:57.6638294Z remote: Counting objects: 14% (613/4376) 2024-06-01T03:42:57.6638985Z remote: Counting objects: 15% (657/4376) 2024-06-01T03:42:57.6639718Z remote: Counting objects: 16% (701/4376) 2024-06-01T03:42:57.6640469Z remote: Counting objects: 17% (744/4376) 2024-06-01T03:42:57.6641224Z remote: Counting objects: 18% (788/4376) 2024-06-01T03:42:57.6641934Z remote: Counting objects: 19% (832/4376) 2024-06-01T03:42:57.6642804Z remote: Counting objects: 20% (876/4376) 2024-06-01T03:42:57.6643406Z remote: Counting objects: 21% (919/4376) 2024-06-01T03:42:57.6644072Z remote: Counting objects: 22% (963/4376) 2024-06-01T03:42:57.6644772Z remote: Counting objects: 23% (1007/4376) 2024-06-01T03:42:57.6645491Z remote: Counting objects: 24% (1051/4376) 2024-06-01T03:42:57.6646208Z remote: Counting objects: 25% (1094/4376) 2024-06-01T03:42:57.6646756Z remote: Counting objects: 26% (1138/4376) 2024-06-01T03:42:57.6647285Z remote: Counting objects: 27% (1182/4376) 2024-06-01T03:42:57.6647968Z remote: Counting objects: 28% (1226/4376) 2024-06-01T03:42:57.6648892Z remote: Counting objects: 29% (1270/4376) 2024-06-01T03:42:57.6649436Z remote: Counting objects: 30% (1313/4376) 2024-06-01T03:42:57.6649993Z remote: Counting objects: 31% (1357/4376) 2024-06-01T03:42:57.6650611Z remote: Counting objects: 32% (1401/4376) 2024-06-01T03:42:57.6651341Z remote: Counting objects: 33% (1445/4376) 2024-06-01T03:42:57.6651932Z remote: Counting objects: 34% (1488/4376) 2024-06-01T03:42:57.6652457Z remote: Counting objects: 35% (1532/4376) 2024-06-01T03:42:57.6653029Z remote: Counting objects: 36% (1576/4376) 2024-06-01T03:42:57.6653740Z remote: Counting objects: 37% (1620/4376) 2024-06-01T03:42:57.6654437Z remote: Counting objects: 38% (1663/4376) 2024-06-01T03:42:57.6654998Z remote: Counting objects: 39% (1707/4376) 2024-06-01T03:42:57.6655528Z remote: Counting objects: 40% (1751/4376) 2024-06-01T03:42:57.6656051Z remote: Counting objects: 41% (1795/4376) 2024-06-01T03:42:57.6656579Z remote: Counting objects: 42% (1838/4376) 2024-06-01T03:42:57.6657264Z remote: Counting objects: 43% (1882/4376) 2024-06-01T03:42:57.6657803Z remote: Counting objects: 44% (1926/4376) 2024-06-01T03:42:57.6658330Z remote: Counting objects: 45% (1970/4376) 2024-06-01T03:42:57.6658857Z remote: Counting objects: 46% (2013/4376) 2024-06-01T03:42:57.6659372Z remote: Counting objects: 47% (2057/4376) 2024-06-01T03:42:57.6659897Z remote: Counting objects: 48% (2101/4376) 2024-06-01T03:42:57.6660424Z remote: Counting objects: 49% (2145/4376) 2024-06-01T03:42:57.6660947Z remote: Counting objects: 50% (2188/4376) 2024-06-01T03:42:57.6661468Z remote: Counting objects: 51% (2232/4376) 2024-06-01T03:42:57.6662001Z remote: Counting objects: 52% (2276/4376) 2024-06-01T03:42:57.6662525Z remote: Counting objects: 53% (2320/4376) 2024-06-01T03:42:57.6663044Z remote: Counting objects: 54% (2364/4376) 2024-06-01T03:42:57.6663612Z remote: Counting objects: 55% (2407/4376) 2024-06-01T03:42:57.6664160Z remote: Counting objects: 56% (2451/4376) 2024-06-01T03:42:57.6664696Z remote: Counting objects: 57% (2495/4376) 2024-06-01T03:42:57.6665219Z remote: Counting objects: 58% (2539/4376) 2024-06-01T03:42:57.6665752Z remote: Counting objects: 59% (2582/4376) 2024-06-01T03:42:57.6666282Z remote: Counting objects: 60% (2626/4376) 2024-06-01T03:42:57.6666811Z remote: Counting objects: 61% (2670/4376) 2024-06-01T03:42:57.6667335Z remote: Counting objects: 62% (2714/4376) 2024-06-01T03:42:57.6667874Z remote: Counting objects: 63% (2757/4376) 2024-06-01T03:42:57.6668402Z remote: Counting objects: 64% (2801/4376) 2024-06-01T03:42:57.6668928Z remote: Counting objects: 65% (2845/4376) 2024-06-01T03:42:57.6669449Z remote: Counting objects: 66% (2889/4376) 2024-06-01T03:42:57.6670204Z remote: Counting objects: 67% (2932/4376) 2024-06-01T03:42:57.6670756Z remote: Counting objects: 68% (2976/4376) 2024-06-01T03:42:57.6671280Z remote: Counting objects: 69% (3020/4376) 2024-06-01T03:42:57.6671810Z remote: Counting objects: 70% (3064/4376) 2024-06-01T03:42:57.6672336Z remote: Counting objects: 71% (3107/4376) 2024-06-01T03:42:57.6672862Z remote: Counting objects: 72% (3151/4376) 2024-06-01T03:42:57.6673385Z remote: Counting objects: 73% (3195/4376) 2024-06-01T03:42:57.6673901Z remote: Counting objects: 74% (3239/4376) 2024-06-01T03:42:57.6674424Z remote: Counting objects: 75% (3282/4376) 2024-06-01T03:42:57.6674948Z remote: Counting objects: 76% (3326/4376) 2024-06-01T03:42:57.6675491Z remote: Counting objects: 77% (3370/4376) 2024-06-01T03:42:57.6676004Z remote: Counting objects: 78% (3414/4376) 2024-06-01T03:42:57.6676526Z remote: Counting objects: 79% (3458/4376) 2024-06-01T03:42:57.6677049Z remote: Counting objects: 80% (3501/4376) 2024-06-01T03:42:57.6677569Z remote: Counting objects: 81% (3545/4376) 2024-06-01T03:42:57.6678085Z remote: Counting objects: 82% (3589/4376) 2024-06-01T03:42:57.6678610Z remote: Counting objects: 83% (3633/4376) 2024-06-01T03:42:57.6679252Z remote: Counting objects: 84% (3676/4376) 2024-06-01T03:42:57.6679771Z remote: Counting objects: 85% (3720/4376) 2024-06-01T03:42:57.6680293Z remote: Counting objects: 86% (3764/4376) 2024-06-01T03:42:57.6680812Z remote: Counting objects: 87% (3808/4376) 2024-06-01T03:42:57.6681333Z remote: Counting objects: 88% (3851/4376) 2024-06-01T03:42:57.6681899Z remote: Counting objects: 89% (3895/4376) 2024-06-01T03:42:57.6682612Z remote: Counting objects: 90% (3939/4376) 2024-06-01T03:42:57.6683143Z remote: Counting objects: 91% (3983/4376) 2024-06-01T03:42:57.6683664Z remote: Counting objects: 92% (4026/4376) 2024-06-01T03:42:57.6684181Z remote: Counting objects: 93% (4070/4376) 2024-06-01T03:42:57.6684702Z remote: Counting objects: 94% (4114/4376) 2024-06-01T03:42:57.6685225Z remote: Counting objects: 95% (4158/4376) 2024-06-01T03:42:57.6685750Z remote: Counting objects: 96% (4201/4376) 2024-06-01T03:42:57.6686354Z remote: Counting objects: 97% (4245/4376) 2024-06-01T03:42:57.6686889Z remote: Counting objects: 98% (4289/4376) 2024-06-01T03:42:57.6687416Z remote: Counting objects: 99% (4333/4376) 2024-06-01T03:42:57.6687938Z remote: Counting objects: 100% (4376/4376) 2024-06-01T03:42:57.6688492Z remote: Counting objects: 100% (4376/4376), done. 2024-06-01T03:42:57.6689063Z remote: Compressing objects: 0% (1/912) 2024-06-01T03:42:57.6689608Z remote: Compressing objects: 1% (10/912) 2024-06-01T03:42:57.6705361Z remote: Compressing objects: 2% (19/912) 2024-06-01T03:42:57.6751149Z remote: Compressing objects: 3% (28/912) 2024-06-01T03:42:57.6753778Z remote: Compressing objects: 4% (37/912) 2024-06-01T03:42:57.6773015Z remote: Compressing objects: 5% (46/912) 2024-06-01T03:42:57.6793529Z remote: Compressing objects: 6% (55/912) 2024-06-01T03:42:57.6806989Z remote: Compressing objects: 7% (64/912) 2024-06-01T03:42:57.6824429Z remote: Compressing objects: 8% (73/912) 2024-06-01T03:42:57.6836633Z remote: Compressing objects: 9% (83/912) 2024-06-01T03:42:57.6848317Z remote: Compressing objects: 10% (92/912) 2024-06-01T03:42:57.6904483Z remote: Compressing objects: 11% (101/912) 2024-06-01T03:42:57.6988967Z remote: Compressing objects: 12% (110/912) 2024-06-01T03:42:57.7053323Z remote: Compressing objects: 13% (119/912) 2024-06-01T03:42:57.7127509Z remote: Compressing objects: 14% (128/912) 2024-06-01T03:42:57.7184538Z remote: Compressing objects: 15% (137/912) 2024-06-01T03:42:57.7268118Z remote: Compressing objects: 16% (146/912) 2024-06-01T03:42:57.7329345Z remote: Compressing objects: 17% (156/912) 2024-06-01T03:42:57.7373410Z remote: Compressing objects: 18% (165/912) 2024-06-01T03:42:57.7436886Z remote: Compressing objects: 19% (174/912) 2024-06-01T03:42:57.7482249Z remote: Compressing objects: 20% (183/912) 2024-06-01T03:42:57.7521647Z remote: Compressing objects: 21% (192/912) 2024-06-01T03:42:57.7556984Z remote: Compressing objects: 22% (201/912) 2024-06-01T03:42:57.7578446Z remote: Compressing objects: 23% (210/912) 2024-06-01T03:42:57.7618569Z remote: Compressing objects: 24% (219/912) 2024-06-01T03:42:57.7648441Z remote: Compressing objects: 25% (228/912) 2024-06-01T03:42:57.7672669Z remote: Compressing objects: 26% (238/912) 2024-06-01T03:42:57.7693620Z remote: Compressing objects: 27% (247/912) 2024-06-01T03:42:57.7712526Z remote: Compressing objects: 28% (256/912) 2024-06-01T03:42:57.7737101Z remote: Compressing objects: 29% (265/912) 2024-06-01T03:42:57.7751088Z remote: Compressing objects: 30% (274/912) 2024-06-01T03:42:57.7764978Z remote: Compressing objects: 31% (283/912) 2024-06-01T03:42:57.7777329Z remote: Compressing objects: 32% (292/912) 2024-06-01T03:42:57.7786256Z remote: Compressing objects: 33% (301/912) 2024-06-01T03:42:57.7794157Z remote: Compressing objects: 34% (311/912) 2024-06-01T03:42:57.7796353Z remote: Compressing objects: 35% (320/912) 2024-06-01T03:42:57.7799264Z remote: Compressing objects: 36% (329/912) 2024-06-01T03:42:57.7801553Z remote: Compressing objects: 37% (338/912) 2024-06-01T03:42:57.7802311Z remote: Compressing objects: 38% (347/912) 2024-06-01T03:42:57.7802864Z remote: Compressing objects: 39% (356/912) 2024-06-01T03:42:57.7804474Z remote: Compressing objects: 40% (365/912) 2024-06-01T03:42:57.7805145Z remote: Compressing objects: 41% (374/912) 2024-06-01T03:42:57.7805692Z remote: Compressing objects: 42% (384/912) 2024-06-01T03:42:57.7809576Z remote: Compressing objects: 43% (393/912) 2024-06-01T03:42:57.7812828Z remote: Compressing objects: 44% (402/912) 2024-06-01T03:42:57.7816706Z remote: Compressing objects: 45% (411/912) 2024-06-01T03:42:57.7820231Z remote: Compressing objects: 46% (420/912) 2024-06-01T03:42:57.7822774Z remote: Compressing objects: 47% (429/912) 2024-06-01T03:42:57.7825567Z remote: Compressing objects: 48% (438/912) 2024-06-01T03:42:57.7828482Z remote: Compressing objects: 49% (447/912) 2024-06-01T03:42:57.7831641Z remote: Compressing objects: 50% (456/912) 2024-06-01T03:42:57.7834639Z remote: Compressing objects: 51% (466/912) 2024-06-01T03:42:57.7838328Z remote: Compressing objects: 52% (475/912) 2024-06-01T03:42:57.7841876Z remote: Compressing objects: 53% (484/912) 2024-06-01T03:42:57.7845343Z remote: Compressing objects: 54% (493/912) 2024-06-01T03:42:57.7847724Z remote: Compressing objects: 55% (502/912) 2024-06-01T03:42:57.7850234Z remote: Compressing objects: 56% (511/912) 2024-06-01T03:42:57.7852664Z remote: Compressing objects: 57% (520/912) 2024-06-01T03:42:57.7855584Z remote: Compressing objects: 58% (529/912) 2024-06-01T03:42:57.7858759Z remote: Compressing objects: 59% (539/912) 2024-06-01T03:42:57.7862387Z remote: Compressing objects: 60% (548/912) 2024-06-01T03:42:57.7864721Z remote: Compressing objects: 61% (557/912) 2024-06-01T03:42:57.7867469Z remote: Compressing objects: 62% (566/912) 2024-06-01T03:42:57.7870134Z remote: Compressing objects: 63% (575/912) 2024-06-01T03:42:57.7872517Z remote: Compressing objects: 64% (584/912) 2024-06-01T03:42:57.7874968Z remote: Compressing objects: 65% (593/912) 2024-06-01T03:42:57.7877680Z remote: Compressing objects: 66% (602/912) 2024-06-01T03:42:57.7878424Z remote: Compressing objects: 67% (612/912) 2024-06-01T03:42:57.7879047Z remote: Compressing objects: 68% (621/912) 2024-06-01T03:42:57.7879590Z remote: Compressing objects: 69% (630/912) 2024-06-01T03:42:57.7880459Z remote: Compressing objects: 70% (639/912) 2024-06-01T03:42:57.7881117Z remote: Compressing objects: 71% (648/912) 2024-06-01T03:42:57.7881659Z remote: Compressing objects: 72% (657/912) 2024-06-01T03:42:57.7882908Z remote: Compressing objects: 73% (666/912) 2024-06-01T03:42:57.7883657Z remote: Compressing objects: 74% (675/912) 2024-06-01T03:42:57.7884224Z remote: Compressing objects: 75% (684/912) 2024-06-01T03:42:57.7884774Z remote: Compressing objects: 76% (694/912) 2024-06-01T03:42:57.7885317Z remote: Compressing objects: 77% (703/912) 2024-06-01T03:42:57.7885868Z remote: Compressing objects: 78% (712/912) 2024-06-01T03:42:57.7886539Z remote: Compressing objects: 79% (721/912) 2024-06-01T03:42:57.7887266Z remote: Compressing objects: 80% (730/912) 2024-06-01T03:42:57.7887876Z remote: Compressing objects: 81% (739/912) 2024-06-01T03:42:57.7888612Z remote: Compressing objects: 82% (748/912) 2024-06-01T03:42:57.7889170Z remote: Compressing objects: 83% (757/912) 2024-06-01T03:42:57.7889710Z remote: Compressing objects: 84% (767/912) 2024-06-01T03:42:57.7890259Z remote: Compressing objects: 85% (776/912) 2024-06-01T03:42:57.7890800Z remote: Compressing objects: 86% (785/912) 2024-06-01T03:42:57.7891435Z remote: Compressing objects: 87% (794/912) 2024-06-01T03:42:57.7891983Z remote: Compressing objects: 88% (803/912) 2024-06-01T03:42:57.7892525Z remote: Compressing objects: 89% (812/912) 2024-06-01T03:42:57.7893614Z remote: Compressing objects: 90% (821/912) 2024-06-01T03:42:57.7896474Z remote: Compressing objects: 91% (830/912) 2024-06-01T03:42:57.7897146Z remote: Compressing objects: 92% (840/912) 2024-06-01T03:42:57.7899282Z remote: Compressing objects: 93% (849/912) 2024-06-01T03:42:57.7899942Z remote: Compressing objects: 94% (858/912) 2024-06-01T03:42:57.7901758Z remote: Compressing objects: 95% (867/912) 2024-06-01T03:42:57.7904227Z remote: Compressing objects: 96% (876/912) 2024-06-01T03:42:57.7904855Z remote: Compressing objects: 97% (885/912) 2024-06-01T03:42:57.7905395Z remote: Compressing objects: 98% (894/912) 2024-06-01T03:42:57.7906450Z remote: Compressing objects: 99% (903/912) 2024-06-01T03:42:57.7906993Z remote: Compressing objects: 100% (912/912) 2024-06-01T03:42:57.7907572Z remote: Compressing objects: 100% (912/912), done. 2024-06-01T03:42:57.7993570Z Receiving objects: 0% (1/29396) 2024-06-01T03:42:57.8014728Z Receiving objects: 1% (294/29396) 2024-06-01T03:42:57.8059555Z Receiving objects: 2% (588/29396) 2024-06-01T03:42:57.8108451Z Receiving objects: 3% (882/29396) 2024-06-01T03:42:57.8160952Z Receiving objects: 4% (1176/29396) 2024-06-01T03:42:57.8262121Z Receiving objects: 5% (1470/29396) 2024-06-01T03:42:57.8284505Z Receiving objects: 6% (1764/29396) 2024-06-01T03:42:57.8414487Z Receiving objects: 7% (2058/29396) 2024-06-01T03:42:57.8445416Z Receiving objects: 8% (2352/29396) 2024-06-01T03:42:57.8478109Z Receiving objects: 9% (2646/29396) 2024-06-01T03:42:57.8508476Z Receiving objects: 10% (2940/29396) 2024-06-01T03:42:57.8538863Z Receiving objects: 11% (3234/29396) 2024-06-01T03:42:57.8569273Z Receiving objects: 12% (3528/29396) 2024-06-01T03:42:57.8607192Z Receiving objects: 13% (3822/29396) 2024-06-01T03:42:57.8637039Z Receiving objects: 14% (4116/29396) 2024-06-01T03:42:57.8666640Z Receiving objects: 15% (4410/29396) 2024-06-01T03:42:57.8702797Z Receiving objects: 16% (4704/29396) 2024-06-01T03:42:57.8741898Z Receiving objects: 17% (4998/29396) 2024-06-01T03:42:57.8783580Z Receiving objects: 18% (5292/29396) 2024-06-01T03:42:57.8903945Z Receiving objects: 19% (5586/29396) 2024-06-01T03:42:57.9031647Z Receiving objects: 20% (5880/29396) 2024-06-01T03:42:57.9068010Z Receiving objects: 21% (6174/29396) 2024-06-01T03:42:57.9094040Z Receiving objects: 22% (6468/29396) 2024-06-01T03:42:57.9110386Z Receiving objects: 23% (6762/29396) 2024-06-01T03:42:57.9132004Z Receiving objects: 24% (7056/29396) 2024-06-01T03:42:57.9161289Z Receiving objects: 25% (7349/29396) 2024-06-01T03:42:57.9172293Z Receiving objects: 26% (7643/29396) 2024-06-01T03:42:57.9182928Z Receiving objects: 27% (7937/29396) 2024-06-01T03:42:57.9200935Z Receiving objects: 28% (8231/29396) 2024-06-01T03:42:57.9823710Z Receiving objects: 29% (8525/29396) 2024-06-01T03:42:57.9831145Z Receiving objects: 30% (8819/29396) 2024-06-01T03:42:57.9874214Z Receiving objects: 31% (9113/29396) 2024-06-01T03:42:58.5039568Z Receiving objects: 32% (9407/29396) 2024-06-01T03:42:58.7923345Z Receiving objects: 33% (9701/29396), 35.57 MiB | 71.13 MiB/s 2024-06-01T03:42:59.4267979Z Receiving objects: 33% (9717/29396), 76.01 MiB | 76.00 MiB/s 2024-06-01T03:42:59.4934174Z Receiving objects: 34% (9995/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.5586136Z Receiving objects: 35% (10289/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.6238495Z Receiving objects: 36% (10583/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.6885433Z Receiving objects: 37% (10877/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.7530215Z Receiving objects: 38% (11171/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.7923445Z Receiving objects: 39% (11465/29396), 115.85 MiB | 77.28 MiB/s 2024-06-01T03:42:59.8186965Z Receiving objects: 39% (11641/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:42:59.8834194Z Receiving objects: 40% (11759/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:42:59.9487684Z Receiving objects: 41% (12053/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:43:00.1658241Z Receiving objects: 42% (12347/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:43:00.2257109Z Receiving objects: 43% (12641/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:43:00.2359635Z Receiving objects: 44% (12935/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:43:00.3670444Z Receiving objects: 45% (13229/29396), 156.31 MiB | 78.19 MiB/s 2024-06-01T03:43:00.6315339Z Receiving objects: 46% (13523/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.6782291Z Receiving objects: 47% (13817/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.6968664Z Receiving objects: 48% (14111/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.6975898Z Receiving objects: 49% (14405/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.6989409Z Receiving objects: 50% (14698/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.6997831Z Receiving objects: 51% (14992/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7006630Z Receiving objects: 52% (15286/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7135877Z Receiving objects: 53% (15580/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7140595Z Receiving objects: 54% (15874/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7145424Z Receiving objects: 55% (16168/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7151947Z Receiving objects: 56% (16462/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7160611Z Receiving objects: 57% (16756/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7170172Z Receiving objects: 58% (17050/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7558803Z Receiving objects: 59% (17344/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7565023Z Receiving objects: 60% (17638/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7574263Z Receiving objects: 61% (17932/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7586092Z Receiving objects: 62% (18226/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7598628Z Receiving objects: 63% (18520/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7607430Z Receiving objects: 64% (18814/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7614471Z Receiving objects: 65% (19108/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7633161Z Receiving objects: 66% (19402/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7638839Z Receiving objects: 67% (19696/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7645793Z Receiving objects: 68% (19990/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7685425Z Receiving objects: 69% (20284/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.7923598Z Receiving objects: 70% (20578/29396), 190.99 MiB | 76.45 MiB/s 2024-06-01T03:43:00.9820732Z Receiving objects: 70% (20588/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9828668Z Receiving objects: 71% (20872/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9836650Z Receiving objects: 72% (21166/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9848363Z Receiving objects: 73% (21460/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9854286Z Receiving objects: 74% (21754/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9983049Z Receiving objects: 75% (22047/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9988155Z Receiving objects: 76% (22341/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:00.9995357Z Receiving objects: 77% (22635/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0000121Z Receiving objects: 78% (22929/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0005999Z Receiving objects: 79% (23223/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0011866Z Receiving objects: 80% (23517/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0121646Z Receiving objects: 81% (23811/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0137253Z Receiving objects: 82% (24105/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0146798Z Receiving objects: 83% (24399/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0164880Z Receiving objects: 84% (24693/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0182744Z Receiving objects: 85% (24987/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0185912Z Receiving objects: 86% (25281/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0190384Z Receiving objects: 87% (25575/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0193647Z Receiving objects: 88% (25869/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0198008Z Receiving objects: 89% (26163/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0201114Z Receiving objects: 90% (26457/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0204540Z Receiving objects: 91% (26751/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0208761Z Receiving objects: 92% (27045/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0218672Z Receiving objects: 93% (27339/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0298185Z Receiving objects: 94% (27633/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0338447Z Receiving objects: 95% (27927/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0371330Z Receiving objects: 96% (28221/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0387491Z Receiving objects: 97% (28515/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0401505Z Receiving objects: 98% (28809/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0432550Z Receiving objects: 99% (29103/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0433494Z remote: Total 29396 (delta 3825), reused 3752 (delta 3462), pack-reused 25020 2024-06-01T03:43:01.0450405Z Receiving objects: 100% (29396/29396), 229.80 MiB | 76.65 MiB/s 2024-06-01T03:43:01.0451108Z Receiving objects: 100% (29396/29396), 244.96 MiB | 75.33 MiB/s, done. 2024-06-01T03:43:01.0491249Z Resolving deltas: 0% (0/15912) 2024-06-01T03:43:01.0515424Z Resolving deltas: 1% (163/15912) 2024-06-01T03:43:01.0522270Z Resolving deltas: 2% (410/15912) 2024-06-01T03:43:01.0531900Z Resolving deltas: 3% (587/15912) 2024-06-01T03:43:01.0551695Z Resolving deltas: 4% (673/15912) 2024-06-01T03:43:01.0573678Z Resolving deltas: 5% (837/15912) 2024-06-01T03:43:01.0588137Z Resolving deltas: 6% (1018/15912) 2024-06-01T03:43:01.0611519Z Resolving deltas: 7% (1118/15912) 2024-06-01T03:43:01.0648011Z Resolving deltas: 8% (1275/15912) 2024-06-01T03:43:01.0682527Z Resolving deltas: 9% (1436/15912) 2024-06-01T03:43:01.0706765Z Resolving deltas: 10% (1627/15912) 2024-06-01T03:43:01.0744630Z Resolving deltas: 11% (1773/15912) 2024-06-01T03:43:01.0754317Z Resolving deltas: 12% (1917/15912) 2024-06-01T03:43:01.0773625Z Resolving deltas: 13% (2069/15912) 2024-06-01T03:43:01.0801695Z Resolving deltas: 14% (2236/15912) 2024-06-01T03:43:01.0814649Z Resolving deltas: 15% (2403/15912) 2024-06-01T03:43:01.0826748Z Resolving deltas: 16% (2551/15912) 2024-06-01T03:43:01.0843172Z Resolving deltas: 17% (2709/15912) 2024-06-01T03:43:01.0854467Z Resolving deltas: 18% (2908/15912) 2024-06-01T03:43:01.0871029Z Resolving deltas: 19% (3025/15912) 2024-06-01T03:43:01.0886738Z Resolving deltas: 20% (3183/15912) 2024-06-01T03:43:01.0900687Z Resolving deltas: 21% (3343/15912) 2024-06-01T03:43:01.0913644Z Resolving deltas: 22% (3508/15912) 2024-06-01T03:43:01.0927642Z Resolving deltas: 23% (3667/15912) 2024-06-01T03:43:01.0939253Z Resolving deltas: 24% (3837/15912) 2024-06-01T03:43:01.0956949Z Resolving deltas: 25% (3981/15912) 2024-06-01T03:43:01.0979186Z Resolving deltas: 26% (4186/15912) 2024-06-01T03:43:01.0996630Z Resolving deltas: 27% (4298/15912) 2024-06-01T03:43:01.1011300Z Resolving deltas: 28% (4462/15912) 2024-06-01T03:43:01.1021886Z Resolving deltas: 29% (4635/15912) 2024-06-01T03:43:01.1027108Z Resolving deltas: 30% (4861/15912) 2024-06-01T03:43:01.1036910Z Resolving deltas: 31% (4934/15912) 2024-06-01T03:43:01.1061171Z Resolving deltas: 32% (5126/15912) 2024-06-01T03:43:01.1071924Z Resolving deltas: 33% (5257/15912) 2024-06-01T03:43:01.1081325Z Resolving deltas: 34% (5411/15912) 2024-06-01T03:43:01.1090035Z Resolving deltas: 35% (5570/15912) 2024-06-01T03:43:01.1097613Z Resolving deltas: 36% (5788/15912) 2024-06-01T03:43:01.1102723Z Resolving deltas: 37% (5908/15912) 2024-06-01T03:43:01.1108730Z Resolving deltas: 38% (6059/15912) 2024-06-01T03:43:01.1116160Z Resolving deltas: 39% (6212/15912) 2024-06-01T03:43:01.1121840Z Resolving deltas: 40% (6398/15912) 2024-06-01T03:43:01.1127048Z Resolving deltas: 41% (6588/15912) 2024-06-01T03:43:01.1141816Z Resolving deltas: 42% (6697/15912) 2024-06-01T03:43:01.1145743Z Resolving deltas: 43% (6949/15912) 2024-06-01T03:43:01.1151267Z Resolving deltas: 44% (7013/15912) 2024-06-01T03:43:01.1157280Z Resolving deltas: 45% (7163/15912) 2024-06-01T03:43:01.1165268Z Resolving deltas: 46% (7328/15912) 2024-06-01T03:43:01.1174851Z Resolving deltas: 47% (7486/15912) 2024-06-01T03:43:01.1186971Z Resolving deltas: 48% (7639/15912) 2024-06-01T03:43:01.1194761Z Resolving deltas: 49% (7797/15912) 2024-06-01T03:43:01.1203205Z Resolving deltas: 50% (7958/15912) 2024-06-01T03:43:01.1222143Z Resolving deltas: 51% (8134/15912) 2024-06-01T03:43:01.1232459Z Resolving deltas: 52% (8282/15912) 2024-06-01T03:43:01.1241881Z Resolving deltas: 53% (8436/15912) 2024-06-01T03:43:01.1248641Z Resolving deltas: 54% (8595/15912) 2024-06-01T03:43:01.1256584Z Resolving deltas: 55% (8753/15912) 2024-06-01T03:43:01.1261367Z Resolving deltas: 56% (8916/15912) 2024-06-01T03:43:01.1268083Z Resolving deltas: 57% (9083/15912) 2024-06-01T03:43:01.1274311Z Resolving deltas: 58% (9247/15912) 2024-06-01T03:43:01.1285440Z Resolving deltas: 59% (9391/15912) 2024-06-01T03:43:01.1291652Z Resolving deltas: 60% (9576/15912) 2024-06-01T03:43:01.1298390Z Resolving deltas: 61% (9719/15912) 2024-06-01T03:43:01.1305088Z Resolving deltas: 62% (9886/15912) 2024-06-01T03:43:01.1316436Z Resolving deltas: 63% (10025/15912) 2024-06-01T03:43:01.1326455Z Resolving deltas: 64% (10188/15912) 2024-06-01T03:43:01.1334075Z Resolving deltas: 65% (10361/15912) 2024-06-01T03:43:01.1344145Z Resolving deltas: 66% (10503/15912) 2024-06-01T03:43:01.1360921Z Resolving deltas: 67% (10662/15912) 2024-06-01T03:43:01.1368681Z Resolving deltas: 68% (10822/15912) 2024-06-01T03:43:01.1380202Z Resolving deltas: 69% (10981/15912) 2024-06-01T03:43:01.1383624Z Resolving deltas: 70% (11247/15912) 2024-06-01T03:43:01.1389550Z Resolving deltas: 71% (11308/15912) 2024-06-01T03:43:01.1397572Z Resolving deltas: 72% (11461/15912) 2024-06-01T03:43:01.1404771Z Resolving deltas: 73% (11617/15912) 2024-06-01T03:43:01.1410693Z Resolving deltas: 74% (11775/15912) 2024-06-01T03:43:01.1419512Z Resolving deltas: 75% (11935/15912) 2024-06-01T03:43:01.1428750Z Resolving deltas: 76% (12135/15912) 2024-06-01T03:43:01.1440548Z Resolving deltas: 77% (12350/15912) 2024-06-01T03:43:01.1453598Z Resolving deltas: 78% (12487/15912) 2024-06-01T03:43:01.1463692Z Resolving deltas: 79% (12629/15912) 2024-06-01T03:43:01.1464168Z Resolving deltas: 80% (12862/15912) 2024-06-01T03:43:01.1472595Z Resolving deltas: 81% (12890/15912) 2024-06-01T03:43:01.1480246Z Resolving deltas: 82% (13074/15912) 2024-06-01T03:43:01.1486563Z Resolving deltas: 83% (13243/15912) 2024-06-01T03:43:01.1493399Z Resolving deltas: 84% (13380/15912) 2024-06-01T03:43:01.1500980Z Resolving deltas: 85% (13528/15912) 2024-06-01T03:43:01.1508558Z Resolving deltas: 86% (13702/15912) 2024-06-01T03:43:01.1522070Z Resolving deltas: 87% (13867/15912) 2024-06-01T03:43:01.1528236Z Resolving deltas: 88% (14056/15912) 2024-06-01T03:43:01.1534223Z Resolving deltas: 89% (14184/15912) 2024-06-01T03:43:01.1541920Z Resolving deltas: 90% (14337/15912) 2024-06-01T03:43:01.1560269Z Resolving deltas: 91% (14496/15912) 2024-06-01T03:43:01.1575317Z Resolving deltas: 92% (14674/15912) 2024-06-01T03:43:01.1588711Z Resolving deltas: 93% (14827/15912) 2024-06-01T03:43:01.1604131Z Resolving deltas: 94% (14958/15912) 2024-06-01T03:43:01.1661091Z Resolving deltas: 95% (15127/15912) 2024-06-01T03:43:01.1667568Z Resolving deltas: 96% (15389/15912) 2024-06-01T03:43:01.1695913Z Resolving deltas: 97% (15450/15912) 2024-06-01T03:43:01.1710575Z Resolving deltas: 98% (15631/15912) 2024-06-01T03:43:01.1739268Z Resolving deltas: 99% (15756/15912) 2024-06-01T03:43:01.1739712Z Resolving deltas: 100% (15912/15912) 2024-06-01T03:43:01.1740165Z Resolving deltas: 100% (15912/15912), done. 2024-06-01T03:43:02.2329450Z + pushd torchbench 2024-06-01T03:43:02.2329868Z ~/workspace/torchbench ~/workspace 2024-06-01T03:43:02.2331486Z + git checkout d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-06-01T03:43:02.2879511Z Note: switching to 'd6015d42d9a1834bc7595c4bd6852562fb80b30b'. 2024-06-01T03:43:02.2880144Z 2024-06-01T03:43:02.2880692Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-01T03:43:02.2881634Z changes and commit them, and you can discard any commits you make in this 2024-06-01T03:43:02.2882574Z state without impacting any branches by switching back to a branch. 2024-06-01T03:43:02.2883072Z 2024-06-01T03:43:02.2883478Z If you want to create a new branch to retain commits you create, you may 2024-06-01T03:43:02.2884492Z do so (now or later) by using -c with the switch command. Example: 2024-06-01T03:43:02.2885117Z 2024-06-01T03:43:02.2885315Z git switch -c 2024-06-01T03:43:02.2885601Z 2024-06-01T03:43:02.2885742Z Or undo this operation with: 2024-06-01T03:43:02.2886003Z 2024-06-01T03:43:02.2886122Z git switch - 2024-06-01T03:43:02.2886309Z 2024-06-01T03:43:02.2886670Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-01T03:43:02.2887179Z 2024-06-01T03:43:02.2887420Z HEAD is now at d6015d42 Add sam_fast torchbench (#2182) 2024-06-01T03:43:02.2887932Z + '[' '' ']' 2024-06-01T03:43:02.2888307Z + python install.py --continue_on_fail 2024-06-01T03:43:04.1482882Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-01T03:43:23.5923621Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-01T03:43:25.9691462Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-01T03:43:46.2906738Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch...OK 2024-06-01T03:43:49.0757956Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-01T03:43:51.6366293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-01T03:43:51.6517409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-01T03:44:00.6849919Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-01T03:44:04.2145780Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-01T03:44:07.7301459Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-01T03:44:11.2529542Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-01T03:44:11.2532993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-01T03:44:13.2781736Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-01T03:44:17.0127462Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-01T03:44:17.0279781Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-01T03:45:10.1598040Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-01T03:45:21.5806875Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-01T03:45:30.3348060Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-01T03:45:38.2407316Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-01T03:45:49.2263090Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-01T03:45:57.4631027Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-01T03:46:04.5159004Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-01T03:46:12.7698382Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-01T03:46:21.5329918Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-01T03:46:29.7943918Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-01T03:46:37.8623196Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-01T03:46:46.1865361Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-01T03:46:50.3709276Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-01T03:47:33.5136874Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-01T03:47:56.0339944Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-01T03:48:04.7419019Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-01T03:48:13.8933713Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-01T03:48:16.3444558Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-01T03:48:19.0521788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-01T03:48:25.8094764Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-01T03:48:34.3030825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-01T03:48:41.3955192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-01T03:48:50.4058260Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-01T03:48:57.8458797Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-01T03:49:04.5304392Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-01T03:49:12.4372262Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-01T03:49:29.2213821Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-01T03:49:36.9334458Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-01T03:49:42.6715659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-01T03:49:49.1312822Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-01T03:49:57.3597918Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-01T03:49:57.3600661Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-01T03:50:11.0950652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-01T03:50:18.7578402Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-01T03:50:18.7580351Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-01T03:50:27.7293665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-01T03:50:30.2617095Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-01T03:50:32.8078287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-01T03:51:12.8276655Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-01T03:52:33.1133100Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-01T03:52:33.1293704Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-01T03:52:35.9097277Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-01T03:52:35.9257169Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-01T03:52:35.9417979Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-01T03:52:35.9574308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-01T03:52:35.9729537Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-01T03:52:35.9893619Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-01T03:52:56.7373054Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-01T03:52:56.7375014Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-01T03:52:59.2628852Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-01T03:53:02.9351213Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-01T03:53:02.9513976Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-01T03:53:02.9676730Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-01T03:53:02.9834150Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-01T03:53:02.9989788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-01T03:53:03.0141895Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-01T03:53:10.3719656Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-01T03:53:13.0775124Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-01T03:53:19.9699262Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-01T03:53:19.9862396Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-01T03:53:20.0024949Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-01T03:53:20.0181838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-01T03:53:20.0337860Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-01T03:53:20.0494166Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-01T03:53:37.1428545Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-01T03:53:55.1247266Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-01T03:53:55.1407365Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-01T03:53:55.1411012Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-01T03:53:55.1415094Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-01T03:53:58.8944006Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-01T03:54:01.9613120Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-01T03:54:01.9770150Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-01T03:54:28.8640142Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-01T03:54:36.1555776Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-01T03:54:42.3988529Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-01T03:55:01.4970473Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-01T03:55:01.5133205Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-01T03:55:01.5294174Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-01T03:55:01.5446915Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-01T03:55:01.5597622Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-01T03:55:01.5750229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-01T03:55:01.5904988Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-01T03:55:01.6058640Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-01T03:55:16.6952872Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-01T03:55:31.8308770Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-01T03:55:31.8474320Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-01T03:55:36.8645785Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-01T03:55:40.2779011Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-01T03:55:40.6784352Z + popd 2024-06-01T03:55:40.6784679Z ~/workspace 2024-06-01T03:55:40.6786621Z + [[ dynamic_inductor_torchbench != *cpu_inductor* ]] 2024-06-01T03:55:40.6787216Z + install_torchrec_and_fbgemm 2024-06-01T03:55:40.6787599Z + local torchrec_commit 2024-06-01T03:55:40.6787964Z ++ get_pinned_commit torchrec 2024-06-01T03:55:40.6788422Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-01T03:55:40.6802849Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T03:55:40.6803599Z + local fbgemm_commit 2024-06-01T03:55:40.6805147Z ++ get_pinned_commit fbgemm 2024-06-01T03:55:40.6805935Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-01T03:55:40.6814914Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-01T03:55:40.6815642Z + pip_uninstall torchrec-nightly 2024-06-01T03:55:40.6816121Z + pip uninstall -y torchrec-nightly 2024-06-01T03:55:41.0336271Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-01T03:55:41.0757359Z + pip_uninstall fbgemm-gpu-nightly 2024-06-01T03:55:41.0757938Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-01T03:55:41.4264365Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-01T03:55:41.4722959Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-01T03:55:41.4723989Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-01T03:55:41.8709807Z Collecting setuptools-git-versioning 2024-06-01T03:55:41.8859727Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-01T03:55:41.9140309Z Collecting scikit-build 2024-06-01T03:55:41.9165474Z Downloading scikit_build-0.17.6-py3-none-any.whl.metadata (14 kB) 2024-06-01T03:55:41.9388516Z Collecting pyre-extensions 2024-06-01T03:55:41.9412724Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-01T03:55:41.9486657Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.0) 2024-06-01T03:55:41.9490070Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-01T03:55:41.9632476Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-01T03:55:41.9658955Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-01T03:55:41.9876873Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-01T03:55:41.9882300Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-01T03:55:41.9887417Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-01T03:55:42.0049214Z Collecting typing-inspect (from pyre-extensions) 2024-06-01T03:55:42.0076745Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-01T03:55:42.0126293Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.0) 2024-06-01T03:55:42.0354221Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-01T03:55:42.0393994Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-01T03:55:42.0469044Z Downloading scikit_build-0.17.6-py3-none-any.whl (84 kB) 2024-06-01T03:55:42.0556621Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-01T03:55:42.0643352Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-01T03:55:42.0723613Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-01T03:55:44.4518452Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-01T03:55:44.5655336Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.17.6 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-01T03:55:44.6904174Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-01T03:55:44.6905434Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-01T03:55:44.6907381Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-01T03:55:45.0329352Z Collecting fbgemm-gpu 2024-06-01T03:55:45.0334580Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-lxwx8490/fbgemm-gpu_41fa251654a545259dda7232f00e3ba0 2024-06-01T03:55:45.0348933Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-lxwx8490/fbgemm-gpu_41fa251654a545259dda7232f00e3ba0 2024-06-01T03:55:45.8184791Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-01T03:55:45.8203361Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-01T03:55:46.2158299Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-01T03:55:46.6230149Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-01T03:55:46.6231761Z Running command git submodule update --init --recursive -q 2024-06-01T03:55:53.9077195Z Preparing metadata (setup.py) ... [?25l- done 2024-06-01T03:55:53.9098149Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.26.4) 2024-06-01T03:55:53.9115769Z Building wheels for collected packages: fbgemm-gpu 2024-06-01T04:37:19.3588926Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-01T04:37:19.5395002Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253252467 sha256=d1268a46968e1d722ea95a88558748aa414a23a952cbc34394221ba1d610790b 2024-06-01T04:37:19.5396762Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-01T04:37:19.5415332Z Successfully built fbgemm-gpu 2024-06-01T04:37:21.8949875Z Installing collected packages: fbgemm-gpu 2024-06-01T04:37:25.0364204Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-01T04:37:25.6552621Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:25.6554208Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:25.9975675Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:25.9980091Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-a8szhvw0 2024-06-01T04:37:25.9998093Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-a8szhvw0 2024-06-01T04:37:26.6224317Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-01T04:37:26.6242160Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:26.8990414Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:27.2221066Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-01T04:37:27.4571472Z Preparing metadata (setup.py) ... [?25l- done 2024-06-01T04:37:27.4620431Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-01T04:37:27.4623226Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.2.2) 2024-06-01T04:37:27.4627727Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-01T04:37:27.4631283Z Requirement already satisfied: torchmetrics in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (1.4.0.post0) 2024-06-01T04:37:27.4634106Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-01T04:37:27.4637080Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-01T04:37:27.4649515Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.26.4) 2024-06-01T04:37:27.4670246Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.8.2) 2024-06-01T04:37:27.5386052Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-01T04:37:27.5390413Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-01T04:37:27.5395770Z Requirement already satisfied: tzdata>=2022.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-01T04:37:27.6172831Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.0) 2024-06-01T04:37:27.6177325Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.4.0a0+gitde352ff) 2024-06-01T04:37:27.6183972Z Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (0.11.2) 2024-06-01T04:37:27.6303689Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-01T04:37:27.6307917Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.0) 2024-06-01T04:37:27.6345163Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-01T04:37:27.6412490Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-01T04:37:27.6416000Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-01T04:37:27.6418553Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-01T04:37:27.6421808Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-01T04:37:27.6424384Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-01T04:37:27.6952678Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-01T04:37:27.7386004Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-01T04:37:27.7473105Z Building wheels for collected packages: torchrec 2024-06-01T04:37:28.1900136Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-01T04:37:28.1910609Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=0cdefe19bf2ff4f156bf673a2d0c10f0c0cb750cf6d63223fe7323a6ebb9f530 2024-06-01T04:37:28.1912196Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-01T04:37:28.1936616Z Successfully built torchrec 2024-06-01T04:37:30.3001501Z Installing collected packages: torchrec 2024-06-01T04:37:30.5137341Z Successfully installed torchrec-0.3.2 2024-06-01T04:37:30.6444618Z ++ pwd 2024-06-01T04:37:30.6446965Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-01T04:37:30.6447523Z + test_dynamo_benchmark torchbench 1 2024-06-01T04:37:30.6450116Z ++ pwd 2024-06-01T04:37:30.6451688Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-01T04:37:30.6453975Z + local suite=torchbench 2024-06-01T04:37:30.6454378Z + shift 2024-06-01T04:37:30.6454678Z + local shard_id=1 2024-06-01T04:37:30.6455129Z + shift 2024-06-01T04:37:30.6455651Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-01T04:37:30.6456338Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-01T04:37:30.6456868Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-01T04:37:30.6457444Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-01T04:37:30.6458315Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-06-01T04:37:30.6469025Z ++ pwd 2024-06-01T04:37:30.6469710Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-01T04:37:30.6470584Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-01T04:37:30.6485029Z + local name=inference 2024-06-01T04:37:30.6485391Z + shift 2024-06-01T04:37:30.6485682Z + local suite=torchbench 2024-06-01T04:37:30.6486082Z + shift 2024-06-01T04:37:30.6486348Z + local shard_id=1 2024-06-01T04:37:30.6486658Z + shift 2024-06-01T04:37:30.6486934Z + partition_flags=() 2024-06-01T04:37:30.6487266Z + local partition_flags 2024-06-01T04:37:30.6487662Z + [[ -n 2 ]] 2024-06-01T04:37:30.6487968Z + [[ -n 1 ]] 2024-06-01T04:37:30.6488581Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-01T04:37:30.6489317Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-01T04:37:30.6489855Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-01T04:37:30.6490389Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-01T04:37:30.6492287Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-01T04:37:36.0795418Z 2024-06-01T04:37:36.5268625Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:37:36.5269172Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:37:36.5273967Z cuda eval lennard_jones 2024-06-01T04:37:40.6802560Z E0601 04:37:40.679000 139887800078976 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00022, (ref-fp64): 0.00059 and shape=torch.Size([4, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:37:40.6803823Z pass 2024-06-01T04:37:40.6815928Z TIMING: code_gen:0.957 inductor_compile:1.18704 backend_compile:2.64967 entire_frame_compile:2.81344 2024-06-01T04:37:40.6817413Z STATS: call_* op count: 18 | FakeTensor.__torch_dispatch__:119 | FakeTensorMode.__torch_dispatch__:1268 | ProxyTorchDispatchMode.__torch_dispatch__:339 2024-06-01T04:37:40.6818564Z Dynamo produced 1 graphs covering 18 ops with 0 graph breaks (0 unique) 2024-06-01T04:37:44.2838757Z 2024-06-01T04:37:44.8000649Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:37:44.8001381Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:37:44.8049982Z cuda eval llama 2024-06-01T04:38:35.2031844Z E0601 04:38:35.202000 140426381050496 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00264, (ref-fp64): 0.00461 and shape=torch.Size([4, 32000]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.000100 2024-06-01T04:38:35.2034498Z pass 2024-06-01T04:38:35.2181200Z TIMING: code_gen:5.38724 inductor_compile:18.85015 backend_compile:42.34477 entire_frame_compile:48.13027 2024-06-01T04:38:35.2182932Z STATS: call_* op count: 1154 | FakeTensor.__torch_dispatch__:3218 | FakeTensorMode.__torch_dispatch__:46756 | attempt fast:1274 | fast is_contiguous:1274 | ProxyTorchDispatchMode.__torch_dispatch__:10235 2024-06-01T04:38:35.2184454Z Dynamo produced 1 graphs covering 1154 ops with 0 graph breaks (0 unique) 2024-06-01T04:38:41.0829371Z 2024-06-01T04:38:41.3588986Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:41.3591973Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:41.3593218Z cuda eval llama_v2_7b_16h 2024-06-01T04:38:41.3595645Z Traceback (most recent call last): 2024-06-01T04:38:41.3596464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T04:38:41.3597352Z ) = runner.load_model( 2024-06-01T04:38:41.3598285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model 2024-06-01T04:38:41.3599088Z benchmark = benchmark_cls( 2024-06-01T04:38:41.3599954Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-01T04:38:41.3600841Z obj = type.__call__(cls, *args, **kwargs) 2024-06-01T04:38:41.3601873Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-06-01T04:38:41.3603716Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:38:41.3605334Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-01T04:38:41.3606577Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:38:41.3607604Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-01T04:38:41.3608379Z self._skip_by_device_name() 2024-06-01T04:38:41.3609206Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-01T04:38:41.3610492Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-01T04:38:41.3611678Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-06-01T04:38:41.3612310Z 2024-06-01T04:38:41.3612437Z model_fail_to_load 2024-06-01T04:38:44.5624393Z 2024-06-01T04:38:44.8381368Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:44.8381964Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:44.8382550Z cuda eval llava 2024-06-01T04:38:44.8386910Z Traceback (most recent call last): 2024-06-01T04:38:44.8387746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T04:38:44.8388663Z ) = runner.load_model( 2024-06-01T04:38:44.8389484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model 2024-06-01T04:38:44.8390600Z benchmark = benchmark_cls( 2024-06-01T04:38:44.8391416Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-01T04:38:44.8392309Z obj = type.__call__(cls, *args, **kwargs) 2024-06-01T04:38:44.8393268Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-01T04:38:44.8394488Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:38:44.8395935Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-01T04:38:44.8397607Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:38:44.8398697Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-01T04:38:44.8399545Z self._skip_by_device_name() 2024-06-01T04:38:44.8400450Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-01T04:38:44.8401859Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-01T04:38:44.8403205Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-01T04:38:44.8403875Z 2024-06-01T04:38:44.8403997Z model_fail_to_load 2024-06-01T04:38:48.0244425Z 2024-06-01T04:38:48.6503849Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:48.6504690Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:48.6512488Z cuda eval maml 2024-06-01T04:38:48.6515354Z pass_due_to_skip 2024-06-01T04:38:48.6523856Z TIMING: 2024-06-01T04:38:48.6524199Z STATS: call_* op count: 0 2024-06-01T04:38:48.6524767Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-01T04:38:51.8907944Z 2024-06-01T04:38:52.3892309Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:52.3893194Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:38:52.3899716Z cuda eval maml_omniglot 2024-06-01T04:38:56.5157736Z E0601 04:38:56.514000 140212529341056 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00035, (ref-fp64): 0.00030 and shape=torch.Size([5, 5]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:38:56.5159465Z pass 2024-06-01T04:38:56.5171705Z TIMING: code_gen:1.20484 inductor_compile:1.44706 backend_compile:2.5978 entire_frame_compile:2.77154 2024-06-01T04:38:56.5173499Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:271 | FakeTensorMode.__torch_dispatch__:2380 | ProxyTorchDispatchMode.__torch_dispatch__:555 2024-06-01T04:38:56.5174662Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-06-01T04:39:00.0889065Z 2024-06-01T04:39:00.3561617Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-06-01T04:39:00.3735238Z 2024-06-01T04:39:00.3735497Z 2024-06-01T04:39:00.4168943Z 0% 0.00/16.9M [00:00 will be ignored 2024-06-01T04:40:14.2561421Z [rank0]:W0601 04:40:14.255000 140580762153600 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-01T04:40:14.2754702Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-01T04:40:14.2759657Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-01T04:40:14.2761319Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-01T04:40:14.2763265Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-01T04:40:14.2764787Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-06-01T04:40:14.2766335Z [rank0]:W0601 04:40:14.274000 140580762153600 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-01T04:40:32.4382667Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 2024-06-01T04:40:32.4383825Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward 2024-06-01T04:40:32.4384828Z self._momentum_update_key_encoder() # update the key encoder 2024-06-01T04:40:32.4386026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context 2024-06-01T04:40:32.4387117Z return func(*args, **kwargs) 2024-06-01T04:40:32.4388099Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder 2024-06-01T04:40:32.4389213Z param_k.mul_(self.m).add_(param_q.mul(1. - self.m)) 2024-06-01T04:40:32.4389578Z 2024-06-01T04:40:33.4365979Z [rank0]:W0601 04:40:33.435000 140580762153600 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program 2024-06-01T04:40:33.4466280Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-06-01T04:40:33.4467552Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-06-01T04:40:33.4468583Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-06-01T04:40:33.4468953Z 2024-06-01T04:40:34.3786065Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-06-01T04:40:34.3787612Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-01T04:40:34.3788792Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] or: 2024-06-01T04:40:34.3790087Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-01T04:40:34.3791497Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] to include these operations in the captured graph. 2024-06-01T04:40:34.3792674Z [rank0]:W0601 04:40:34.378000 140580762153600 torch/_dynamo/variables/tensor.py:715] [5/0] 2024-06-01T04:40:45.7366132Z [rank0]:E0601 04:40:45.735000 140580762153600 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00569, (ref-fp64): 0.00548 and shape=torch.Size([4, 32001]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:40:45.7372489Z pass 2024-06-01T04:40:45.7375610Z TIMING: entire_frame_compile:36.21495 code_gen:8.9494 inductor_compile:16.6808 backend_compile:26.91339 2024-06-01T04:40:45.7376977Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:10623 | FakeTensorMode.__torch_dispatch__:58178 | ProxyTorchDispatchMode.__torch_dispatch__:12396 2024-06-01T04:40:45.7378138Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-06-01T04:40:50.4515443Z 2024-06-01T04:40:50.7283828Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:40:50.7284312Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:40:50.7284754Z cuda eval moondream 2024-06-01T04:40:50.7290072Z Traceback (most recent call last): 2024-06-01T04:40:50.7290806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T04:40:50.7291731Z ) = runner.load_model( 2024-06-01T04:40:50.7292592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model 2024-06-01T04:40:50.7293851Z benchmark = benchmark_cls( 2024-06-01T04:40:50.7294769Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-01T04:40:50.7295774Z obj = type.__call__(cls, *args, **kwargs) 2024-06-01T04:40:50.7296680Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-06-01T04:40:50.7297880Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:40:50.7299198Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-01T04:40:50.7300430Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T04:40:50.7301440Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-01T04:40:50.7302212Z self._skip_by_device_name() 2024-06-01T04:40:50.7303191Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-01T04:40:50.7304470Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-01T04:40:50.7305619Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-06-01T04:40:50.7306280Z 2024-06-01T04:40:50.7306405Z model_fail_to_load 2024-06-01T04:40:53.9379954Z 2024-06-01T04:40:55.8208150Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-01T04:40:56.0259522Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-01T04:40:56.0260439Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-01T04:40:56.0262536Z using fused AdamW: True 2024-06-01T04:40:56.3271717Z 2024-06-01T04:40:56.3272195Z loading model: 0it [00:02, ?it/s] 2024-06-01T04:40:56.3319784Z cuda eval nanogpt 2024-06-01T04:41:32.1159618Z E0601 04:41:32.115000 140023134950016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00437, (ref-fp64): 0.00566 and shape=torch.Size([4, 1, 50304]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:41:32.1161205Z pass 2024-06-01T04:41:32.1172806Z TIMING: code_gen:2.97667 inductor_compile:10.47897 backend_compile:27.89871 entire_frame_compile:33.66357 2024-06-01T04:41:32.1175516Z STATS: call_* op count: 834 | FakeTensorMode.__torch_dispatch__:40266 | FakeTensor.__torch_dispatch__:2657 | attempt fast:2347 | fast is_contiguous:2347 | ProxyTorchDispatchMode.__torch_dispatch__:8284 2024-06-01T04:41:32.1176901Z Dynamo produced 1 graphs covering 834 ops with 0 graph breaks (0 unique) 2024-06-01T04:41:37.2512128Z 2024-06-01T04:41:39.6128931Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:41:39.6129459Z loading model: 0it [00:02, ?it/s] 2024-06-01T04:41:39.6132713Z cuda eval nvidia_deeprecommender 2024-06-01T04:41:44.3439158Z E0601 04:41:44.342000 140398439150208 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00066, (ref-fp64): 0.00062 and shape=torch.Size([4, 197951]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:41:44.3440389Z pass 2024-06-01T04:41:44.3451016Z TIMING: code_gen:0.7192 inductor_compile:0.96095 backend_compile:3.09632 entire_frame_compile:3.28806 2024-06-01T04:41:44.3452373Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1793 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170 2024-06-01T04:41:44.3453516Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-06-01T04:41:47.8845373Z 2024-06-01T04:41:48.8281856Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:41:48.8282459Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:41:48.8303497Z cuda eval opacus_cifar10 2024-06-01T04:42:00.2373177Z E0601 04:42:00.236000 140152535409280 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00864, (ref-fp64): 0.01055 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:42:00.2374693Z pass 2024-06-01T04:42:00.2377072Z TIMING: code_gen:2.88048 inductor_compile:4.30683 backend_compile:6.66532 entire_frame_compile:9.87479 2024-06-01T04:42:00.2378415Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:7448 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866 2024-06-01T04:42:00.2379597Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-06-01T04:42:04.0163948Z 2024-06-01T04:42:04.6349105Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:04.6349580Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:04.6419505Z cuda eval phlippe_densenet 2024-06-01T04:42:19.6034212Z E0601 04:42:19.602000 140596136153728 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00077, (ref-fp64): 0.00070 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:42:19.6037333Z pass 2024-06-01T04:42:19.6047362Z TIMING: code_gen:3.93591 inductor_compile:6.81742 backend_compile:11.36516 entire_frame_compile:13.25804 2024-06-01T04:42:19.6048998Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:3838 | FakeTensorMode.__torch_dispatch__:28866 | ProxyTorchDispatchMode.__torch_dispatch__:7588 2024-06-01T04:42:19.6050158Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-06-01T04:42:23.5387149Z 2024-06-01T04:42:24.0594316Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:24.0595051Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:24.0624231Z cuda eval phlippe_resnet 2024-06-01T04:42:30.9290186Z E0601 04:42:30.928000 139966309245568 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00838, (ref-fp64): 0.00383 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:42:30.9291555Z pass 2024-06-01T04:42:30.9305309Z TIMING: code_gen:1.46217 inductor_compile:2.50575 backend_compile:4.74557 entire_frame_compile:5.39723 2024-06-01T04:42:30.9306742Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1447 | FakeTensorMode.__torch_dispatch__:10983 | ProxyTorchDispatchMode.__torch_dispatch__:2856 2024-06-01T04:42:30.9308064Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-01T04:42:34.5304355Z 2024-06-01T04:42:34.8155135Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:34.8155697Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:34.8156179Z cuda eval pyhpc_equation_of_state 2024-06-01T04:42:41.5885791Z E0601 04:42:41.587000 140300751823488 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03091, (ref-fp64): 1242.88896 and shape=torch.Size([4, 4, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:42:41.5887289Z pass 2024-06-01T04:42:41.5890226Z TIMING: code_gen:2.31285 inductor_compile:3.56463 backend_compile:5.09048 entire_frame_compile:5.48844 2024-06-01T04:42:41.5891655Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:10370 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061 2024-06-01T04:42:41.5892839Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-06-01T04:42:45.1969474Z 2024-06-01T04:42:45.4893334Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:45.4894027Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:42:45.4895328Z cuda eval pyhpc_isoneutral_mixing 2024-06-01T04:43:12.2109098Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 2024-06-01T04:43:12.2110473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in run_n_iterations 2024-06-01T04:43:12.2111362Z return self.model_iter_fn(mod, inputs, collect_outputs=True) 2024-06-01T04:43:12.2112250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 430, in forward_pass 2024-06-01T04:43:12.2112989Z return mod(*inputs) 2024-06-01T04:43:12.2114151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T04:43:12.2115225Z return forward_call(*args, **kwargs) 2024-06-01T04:43:12.2116516Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward 2024-06-01T04:43:12.2117866Z return isoneutral_pytorch.isoneutral_diffusion_pre( 2024-06-01T04:43:12.2119148Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre 2024-06-01T04:43:12.2120697Z K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :]) 2024-06-01T04:43:12.2121068Z 2024-06-01T04:43:12.3359160Z pass 2024-06-01T04:43:12.3360088Z TIMING: code_gen:5.98357 inductor_compile:15.96412 backend_compile:24.12219 entire_frame_compile:25.72165 2024-06-01T04:43:12.3361645Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:55945 | FakeTensor.__torch_dispatch__:5080 | ProxyTorchDispatchMode.__torch_dispatch__:14005 2024-06-01T04:43:12.3363032Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-06-01T04:43:16.9331906Z 2024-06-01T04:43:17.6680227Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-01T04:43:17.7489806Z 2024-06-01T04:43:17.7490233Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:43:17.7491301Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-01T04:43:17.7492148Z cuda eval pyhpc_turbulent_kinetic_energy 2024-06-01T04:43:17.7947048Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-01T04:43:46.6656613Z pass 2024-06-01T04:43:46.6759846Z TIMING: code_gen:9.66253 inductor_compile:16.27241 backend_compile:25.55445 entire_frame_compile:27.3202 2024-06-01T04:43:46.6761551Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:49774 | FakeTensor.__torch_dispatch__:2561 | ProxyTorchDispatchMode.__torch_dispatch__:14635 2024-06-01T04:43:46.6762862Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-06-01T04:43:51.0383325Z 2024-06-01T04:43:51.8545241Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:43:51.8547487Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:43:51.8558991Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-06-01T04:44:10.1581729Z E0601 04:44:10.157000 140187530863232 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01044, (ref-fp64): 0.01069 and shape=torch.Size([1, 3, 256, 256]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:44:10.1583125Z pass 2024-06-01T04:44:10.1617067Z TIMING: code_gen:3.62629 inductor_compile:5.60118 backend_compile:7.62401 entire_frame_compile:8.19294 2024-06-01T04:44:10.1618410Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:11393 | FakeTensor.__torch_dispatch__:1335 | ProxyTorchDispatchMode.__torch_dispatch__:3743 2024-06-01T04:44:10.1619624Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-06-01T04:44:13.9600166Z 2024-06-01T04:44:15.0764377Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:44:15.0764974Z loading model: 0it [00:01, ?it/s] 2024-06-01T04:44:15.0789206Z cuda eval pytorch_stargan 2024-06-01T04:44:44.3003726Z pass 2024-06-01T04:44:44.3066470Z TIMING: code_gen:2.51659 inductor_compile:4.85798 backend_compile:7.38982 entire_frame_compile:8.00811 2024-06-01T04:44:44.3070478Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:17669 | FakeTensor.__torch_dispatch__:1566 | ProxyTorchDispatchMode.__torch_dispatch__:4659 2024-06-01T04:44:44.3072348Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-06-01T04:44:48.1091791Z 2024-06-01T04:44:48.7476012Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:44:48.7476992Z loading model: 0it [00:00, ?it/s] 2024-06-01T04:44:48.7498964Z cuda eval pytorch_unet 2024-06-01T04:45:10.3502022Z E0601 04:45:10.349000 139667175322240 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00020, (ref-fp64): 0.00020 and shape=torch.Size([2, 2, 640, 959]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T04:45:10.3505314Z pass 2024-06-01T04:45:10.3629900Z TIMING: code_gen:4.41902 inductor_compile:6.43096 backend_compile:8.50927 entire_frame_compile:9.35893 2024-06-01T04:45:10.3632034Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1807 | FakeTensorMode.__torch_dispatch__:14774 | ProxyTorchDispatchMode.__torch_dispatch__:4409 2024-06-01T04:45:10.3633642Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-01T04:45:14.1327104Z 2024-06-01T04:45:14.9721281Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-06-01T04:45:14.9883121Z 2024-06-01T04:45:14.9883335Z 2024-06-01T04:45:15.0888307Z 0% 0.00/230M [00:00 2024-06-01T04:53:48.5724016Z torchbench_main() 2024-06-01T04:53:48.5724715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 452, in torchbench_main 2024-06-01T04:53:48.5725535Z main(TorchBenchmarkRunner(), original_dir) 2024-06-01T04:53:48.5726622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3660, in main 2024-06-01T04:53:48.5728696Z process_entry(0, runner, original_dir, args) 2024-06-01T04:53:48.5729478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3592, in process_entry 2024-06-01T04:53:48.5732279Z return run(runner, args, original_dir) 2024-06-01T04:53:48.5733059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4193, in run 2024-06-01T04:53:48.5736687Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-01T04:53:48.5737425Z AssertionError: nothing in example_inputs had a dim with 4 2024-06-01T04:53:51.5365122Z Run failed with return code: 1 2024-06-01T04:53:51.5365680Z Output: None 2024-06-01T04:53:51.5365981Z Error: None 2024-06-01T04:53:54.1173377Z 2024-06-01T04:53:54.3506177Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-06-01T04:53:54.3691809Z 2024-06-01T04:53:54.3692090Z 2024-06-01T04:53:54.3923650Z 0% 0.00/8.79M [00:00 2024-06-01T04:59:21.6869420Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-01T04:59:21.6870326Z 2024-06-01T04:59:22.7473343Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] Graph break from `Tensor.item()`, consider setting: 2024-06-01T04:59:22.7474977Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-01T04:59:22.7476129Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] or: 2024-06-01T04:59:22.7477243Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-01T04:59:22.7478855Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] to include these operations in the captured graph. 2024-06-01T04:59:22.7479962Z W0601 04:59:22.746000 140291923030656 torch/_dynamo/variables/tensor.py:715] [13/0] 2024-06-01T05:00:06.8825122Z pass 2024-06-01T05:00:06.8825856Z TIMING: entire_frame_compile:66.22966 code_gen:9.33697 inductor_compile:21.74403 backend_compile:36.53744 2024-06-01T05:00:06.8830746Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:87337 | FakeTensor.__torch_dispatch__:7899 | ProxyTorchDispatchMode.__torch_dispatch__:18376 | attempt fast:428 | fast is_contiguous:284 | slow both tensors nontrivially broadcast:144 2024-06-01T05:00:06.8832575Z Dynamo produced 25 graphs covering 1358 ops with 17 graph breaks (2 unique) 2024-06-01T05:00:11.9107331Z 2024-06-01T05:00:13.9347800Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:00:13.9348699Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:00:13.9442727Z cuda eval yolov3 2024-06-01T05:00:37.8369170Z W0601 05:00:37.836000 140669941265024 torch/_inductor/utils.py:1189] [7/0] DeviceCopy in input program 2024-06-01T05:00:38.0177985Z W0601 05:00:38.017000 140669941265024 torch/_inductor/utils.py:1189] [7/0] DeviceCopy in input program 2024-06-01T05:00:39.4472333Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg1_1). Found from : 2024-06-01T05:00:39.4473564Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-01T05:00:39.4474663Z self.create_grids((nx, ny), p.device) 2024-06-01T05:00:39.4476372Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-01T05:00:39.4477360Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-01T05:00:39.4477714Z 2024-06-01T05:00:41.5908610Z W0601 05:00:41.590000 140669941265024 torch/_inductor/utils.py:1189] [7/1] DeviceCopy in input program 2024-06-01T05:00:41.7455484Z W0601 05:00:41.745000 140669941265024 torch/_inductor/utils.py:1189] [7/1] DeviceCopy in input program 2024-06-01T05:00:43.2796840Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg3_1). Found from : 2024-06-01T05:00:43.2798119Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-01T05:00:43.2799038Z self.create_grids((nx, ny), p.device) 2024-06-01T05:00:43.2799982Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-01T05:00:43.2800938Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-01T05:00:43.2801292Z 2024-06-01T05:01:22.4059032Z E0601 05:01:22.404000 140669941265024 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01904, (ref-fp64): 0.01998 and shape=torch.Size([4, 12096, 85]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100 2024-06-01T05:01:22.4071040Z pass 2024-06-01T05:01:22.4150703Z TIMING: entire_frame_compile:54.30737 code_gen:12.06904 inductor_compile:14.33239 backend_compile:16.96333 2024-06-01T05:01:22.4155011Z STATS: call_* op count: 120 | FakeTensor.__torch_dispatch__:3116 | FakeTensorMode.__torch_dispatch__:18519 | ProxyTorchDispatchMode.__torch_dispatch__:895 | attempt fast:51 | fast is_contiguous:39 | slow no contiguity match:12 2024-06-01T05:01:22.4158060Z Dynamo produced 27 graphs covering 120 ops with 2 graph breaks (1 unique) 2024-06-01T05:01:23.9046095Z accuracy pass_rate=82.61% 2024-06-01T05:01:23.9049614Z calls_captured gmean=0.00x mean=506.391x 2024-06-01T05:01:23.9053065Z unique_graphs gmean=0.00x mean=2.217x 2024-06-01T05:01:23.9056417Z graph_breaks gmean=0.00x mean=0.783x 2024-06-01T05:01:23.9059972Z unique_graph_breaks gmean=0.00x mean=0.196x 2024-06-01T05:01:23.9063259Z autograd_captures gmean=0.00x mean=0.000x 2024-06-01T05:01:23.9066600Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-01T05:01:23.9070352Z cudagraph_skips gmean=0.00x mean=0.196x 2024-06-01T05:01:24.7193295Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/dynamic_inductor_torchbench_inference.csv 2024-06-01T05:01:25.0399559Z lennard_jones PASS 2024-06-01T05:01:25.0402988Z llama PASS 2024-06-01T05:01:25.0408125Z llama_v2_7b_16h XFAIL 2024-06-01T05:01:25.0413660Z llava XFAIL 2024-06-01T05:01:25.0418828Z maml XFAIL 2024-06-01T05:01:25.0424097Z maml_omniglot PASS 2024-06-01T05:01:25.0429334Z mnasnet1_0 PASS 2024-06-01T05:01:25.0434917Z mobilenet_v2 PASS 2024-06-01T05:01:25.0440046Z mobilenet_v2_quantized_qat XFAIL 2024-06-01T05:01:25.0445667Z mobilenet_v3_large PASS 2024-06-01T05:01:25.0450774Z moco PASS 2024-06-01T05:01:25.0456135Z moondream XFAIL 2024-06-01T05:01:25.0461335Z nanogpt PASS 2024-06-01T05:01:25.0466550Z nvidia_deeprecommender PASS 2024-06-01T05:01:25.0472183Z opacus_cifar10 PASS 2024-06-01T05:01:25.0477518Z phlippe_densenet PASS 2024-06-01T05:01:25.0482724Z phlippe_resnet PASS 2024-06-01T05:01:25.0488097Z pyhpc_equation_of_state PASS 2024-06-01T05:01:25.0493301Z pyhpc_isoneutral_mixing PASS 2024-06-01T05:01:25.0498491Z pyhpc_turbulent_kinetic_energy PASS 2024-06-01T05:01:25.0503766Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-01T05:01:25.0509002Z pytorch_stargan PASS 2024-06-01T05:01:25.0514425Z pytorch_unet PASS 2024-06-01T05:01:25.0519523Z resnet152 PASS 2024-06-01T05:01:25.0524937Z resnet18 PASS 2024-06-01T05:01:25.0530235Z resnet50 PASS 2024-06-01T05:01:25.0535301Z resnet50_quantized_qat XFAIL 2024-06-01T05:01:25.0540566Z resnext50_32x4d PASS 2024-06-01T05:01:25.0545724Z sam PASS 2024-06-01T05:01:25.0551043Z shufflenet_v2_x1_0 PASS 2024-06-01T05:01:25.0556482Z soft_actor_critic PASS 2024-06-01T05:01:25.0561662Z speech_transformer PASS 2024-06-01T05:01:25.0567173Z squeezenet1_1 PASS 2024-06-01T05:01:25.0572356Z stable_diffusion_text_encoder PASS 2024-06-01T05:01:25.0577501Z stable_diffusion_unet XFAIL 2024-06-01T05:01:25.0582733Z timm_efficientnet PASS 2024-06-01T05:01:25.0587901Z timm_regnet PASS 2024-06-01T05:01:25.0593318Z timm_resnest PASS 2024-06-01T05:01:25.0598314Z timm_vision_transformer PASS 2024-06-01T05:01:25.0603664Z timm_vision_transformer_large XFAIL 2024-06-01T05:01:25.0608908Z timm_vovnet PASS 2024-06-01T05:01:25.0614075Z torch_multimodal_clip PASS 2024-06-01T05:01:25.0619206Z tts_angular PASS 2024-06-01T05:01:25.0624398Z vgg16 PASS 2024-06-01T05:01:25.0629600Z vision_maskrcnn PASS 2024-06-01T05:01:25.0635054Z yolov3 PASS 2024-06-01T05:01:25.1094494Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/dynamic_inductor_torchbench_inference.csv 2024-06-01T05:01:25.4240887Z lennard_jones PASS 2024-06-01T05:01:25.4244514Z llama PASS 2024-06-01T05:01:25.4249696Z llama_v2_7b_16h PASS 2024-06-01T05:01:25.4254609Z llava PASS 2024-06-01T05:01:25.4259865Z maml PASS 2024-06-01T05:01:25.4264772Z maml_omniglot PASS 2024-06-01T05:01:25.4269733Z mnasnet1_0 PASS 2024-06-01T05:01:25.4274762Z mobilenet_v2 PASS 2024-06-01T05:01:25.4279941Z mobilenet_v2_quantized_qat PASS 2024-06-01T05:01:25.4284977Z mobilenet_v3_large PASS 2024-06-01T05:01:25.4290062Z moco PASS 2024-06-01T05:01:25.4294997Z moondream PASS 2024-06-01T05:01:25.4299965Z nanogpt PASS 2024-06-01T05:01:25.4305201Z nvidia_deeprecommender PASS 2024-06-01T05:01:25.4310803Z opacus_cifar10 PASS 2024-06-01T05:01:25.4315976Z phlippe_densenet PASS 2024-06-01T05:01:25.4320993Z phlippe_resnet PASS 2024-06-01T05:01:25.4326392Z pyhpc_equation_of_state PASS 2024-06-01T05:01:25.4331572Z pyhpc_isoneutral_mixing PASS 2024-06-01T05:01:25.4336549Z pyhpc_turbulent_kinetic_energy PASS 2024-06-01T05:01:25.4341524Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-01T05:01:25.4346541Z pytorch_stargan PASS 2024-06-01T05:01:25.4351831Z pytorch_unet PASS 2024-06-01T05:01:25.4356764Z resnet152 PASS 2024-06-01T05:01:25.4361787Z resnet18 PASS 2024-06-01T05:01:25.4367328Z resnet50 PASS 2024-06-01T05:01:25.4372433Z resnet50_quantized_qat PASS 2024-06-01T05:01:25.4377409Z resnext50_32x4d PASS 2024-06-01T05:01:25.4382518Z sam PASS 2024-06-01T05:01:25.4387486Z shufflenet_v2_x1_0 PASS 2024-06-01T05:01:25.4392720Z soft_actor_critic PASS 2024-06-01T05:01:25.4397762Z speech_transformer PASS 2024-06-01T05:01:25.4402904Z squeezenet1_1 PASS 2024-06-01T05:01:25.4407905Z stable_diffusion_text_encoder PASS 2024-06-01T05:01:25.4412959Z stable_diffusion_unet PASS 2024-06-01T05:01:25.4417884Z timm_efficientnet PASS 2024-06-01T05:01:25.4422931Z timm_regnet PASS 2024-06-01T05:01:25.4427922Z timm_resnest PASS 2024-06-01T05:01:25.4433042Z timm_vision_transformer PASS 2024-06-01T05:01:25.4438131Z timm_vision_transformer_large PASS 2024-06-01T05:01:25.4443320Z timm_vovnet PASS 2024-06-01T05:01:25.4448317Z torch_multimodal_clip PASS 2024-06-01T05:01:25.4453507Z tts_angular PASS 2024-06-01T05:01:25.4458543Z vgg16 PASS 2024-06-01T05:01:25.4464036Z vision_maskrcnn PASS 2024-06-01T05:01:25.4469102Z yolov3 PASS 2024-06-01T05:01:25.4935265Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-06-01T05:01:25.4935863Z ++ pwd 2024-06-01T05:01:25.4936829Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-01T05:01:25.4938774Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-01T05:01:25.4947392Z + local name=training 2024-06-01T05:01:25.4947871Z + shift 2024-06-01T05:01:25.4948269Z + local suite=torchbench 2024-06-01T05:01:25.4948672Z + shift 2024-06-01T05:01:25.4948953Z + local shard_id=1 2024-06-01T05:01:25.4952944Z + shift 2024-06-01T05:01:25.4953389Z + partition_flags=() 2024-06-01T05:01:25.4953852Z + local partition_flags 2024-06-01T05:01:25.4954405Z + [[ -n 2 ]] 2024-06-01T05:01:25.4954810Z + [[ -n 1 ]] 2024-06-01T05:01:25.4955434Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-01T05:01:25.4956183Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-01T05:01:25.4956728Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-01T05:01:25.4957249Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-01T05:01:25.4959326Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-06-01T05:01:30.7943434Z 2024-06-01T05:01:31.2293226Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:31.2293711Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:31.2294156Z cuda train lennard_jones 2024-06-01T05:01:37.2618441Z W0601 05:01:37.260000 140631360848512 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:01:41.8320250Z pass 2024-06-01T05:01:41.8321059Z TIMING: entire_frame_compile:8.67965 code_gen:4.55202 inductor_compile:6.1422 backend_compile:7.92149 2024-06-01T05:01:41.8322631Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:830 | FakeTensorMode.__torch_dispatch__:5938 | attempt fast:82 | fast is_contiguous:82 | ProxyTorchDispatchMode.__torch_dispatch__:1221 2024-06-01T05:01:41.8323993Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-01T05:01:45.4901832Z 2024-06-01T05:01:45.7661949Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:45.7662642Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:45.7663266Z cuda train llava 2024-06-01T05:01:45.7667527Z Traceback (most recent call last): 2024-06-01T05:01:45.7668428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T05:01:45.7669231Z ) = runner.load_model( 2024-06-01T05:01:45.7669915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 303, in load_model 2024-06-01T05:01:45.7671239Z benchmark = benchmark_cls( 2024-06-01T05:01:45.7672177Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-01T05:01:45.7673242Z obj = type.__call__(cls, *args, **kwargs) 2024-06-01T05:01:45.7674157Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-01T05:01:45.7675380Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T05:01:45.7676677Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-01T05:01:45.7679328Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-01T05:01:45.7680343Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-01T05:01:45.7681112Z self._skip_by_device_name() 2024-06-01T05:01:45.7681952Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-01T05:01:45.7683374Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-01T05:01:45.7684558Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-01T05:01:45.7685385Z 2024-06-01T05:01:45.7685505Z model_fail_to_load 2024-06-01T05:01:48.9521397Z 2024-06-01T05:01:49.5397588Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:49.5398214Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:01:49.5398713Z cuda train maml_omniglot 2024-06-01T05:01:59.7337913Z W0601 05:01:59.732000 139816516067968 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:02:05.8387346Z E0601 05:02:05.837000 139816516067968 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00019, (ref-fp64): 0.00019 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:02:05.8399092Z E0601 05:02:05.839000 139816516067968 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00021, (ref-fp64): 0.00021 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:02:05.8411500Z E0601 05:02:05.840000 139816516067968 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00118, (ref-fp64): 0.00118 and shape=torch.Size([64, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:02:05.8427791Z E0601 05:02:05.842000 139816516067968 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00094, (ref-fp64): 0.00077 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:02:05.8440465Z E0601 05:02:05.843000 139816516067968 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00009, (ref-fp64): 0.00014 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:02:05.8469794Z pass 2024-06-01T05:02:05.8470987Z TIMING: entire_frame_compile:14.17762 code_gen:7.80098 inductor_compile:9.81818 backend_compile:13.09762 2024-06-01T05:02:05.8472524Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1199 | FakeTensorMode.__torch_dispatch__:7806 | attempt fast:91 | fast is_contiguous:91 | ProxyTorchDispatchMode.__torch_dispatch__:1538 2024-06-01T05:02:05.8473858Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-01T05:02:09.7301624Z 2024-06-01T05:02:12.4034231Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:02:12.4035409Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:02:12.4036584Z cuda train mnasnet1_0 2024-06-01T05:02:56.9603027Z W0601 05:02:56.959000 139711399002752 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:03:41.1120353Z E0601 05:03:41.111000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 17.89697, (ref-fp64): 16.10112 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1126308Z E0601 05:03:41.112000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.62892, (ref-fp64): 2.63768 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1137115Z E0601 05:03:41.113000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00380, (ref-fp64): 0.00317 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1141600Z E0601 05:03:41.113000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00397, (ref-fp64): 0.01071 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1145747Z E0601 05:03:41.114000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02171, (ref-fp64): 0.04361 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1150149Z E0601 05:03:41.114000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07508, (ref-fp64): 0.31671 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1154988Z E0601 05:03:41.115000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02484, (ref-fp64): 0.03095 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1159228Z E0601 05:03:41.115000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04648, (ref-fp64): 0.12865 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1163598Z E0601 05:03:41.115000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07230, (ref-fp64): 0.11725 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1168190Z E0601 05:03:41.116000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00822, (ref-fp64): 0.01888 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1172478Z E0601 05:03:41.116000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03158, (ref-fp64): 0.07048 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1176838Z E0601 05:03:41.117000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05643, (ref-fp64): 0.08874 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1181248Z E0601 05:03:41.117000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01707, (ref-fp64): 0.03285 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1185645Z E0601 05:03:41.118000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02978, (ref-fp64): 0.07691 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1189908Z E0601 05:03:41.118000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05554, (ref-fp64): 0.09192 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1195087Z E0601 05:03:41.119000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00726, (ref-fp64): 0.01301 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1199308Z E0601 05:03:41.119000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02001, (ref-fp64): 0.05731 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1203815Z E0601 05:03:41.119000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03207, (ref-fp64): 0.06193 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1208252Z E0601 05:03:41.120000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01536, (ref-fp64): 0.01863 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1212552Z E0601 05:03:41.120000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01570, (ref-fp64): 0.05527 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1216944Z E0601 05:03:41.121000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02177, (ref-fp64): 0.04765 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1221609Z E0601 05:03:41.121000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01264, (ref-fp64): 0.01822 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1225954Z E0601 05:03:41.122000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02933, (ref-fp64): 0.08167 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1230367Z E0601 05:03:41.122000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06520, (ref-fp64): 0.12304 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1235191Z E0601 05:03:41.123000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01719 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1239374Z E0601 05:03:41.123000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03691, (ref-fp64): 0.07127 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1243717Z E0601 05:03:41.123000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.08089, (ref-fp64): 0.10195 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1248410Z E0601 05:03:41.124000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01733, (ref-fp64): 0.02330 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1252555Z E0601 05:03:41.124000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01548, (ref-fp64): 0.04592 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1256785Z E0601 05:03:41.125000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05907, (ref-fp64): 0.07264 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1261403Z E0601 05:03:41.125000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01877, (ref-fp64): 0.02265 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1265643Z E0601 05:03:41.126000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02453, (ref-fp64): 0.07163 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1270115Z E0601 05:03:41.126000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.13266, (ref-fp64): 0.14894 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1274809Z E0601 05:03:41.127000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03148, (ref-fp64): 0.03128 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1278928Z E0601 05:03:41.127000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04584, (ref-fp64): 0.09904 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1283462Z E0601 05:03:41.127000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.11640, (ref-fp64): 0.11506 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1288032Z E0601 05:03:41.128000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06361, (ref-fp64): 0.06794 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1292525Z E0601 05:03:41.128000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03786, (ref-fp64): 0.06575 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1296535Z E0601 05:03:41.129000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07684, (ref-fp64): 0.06909 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1301118Z E0601 05:03:41.129000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03423, (ref-fp64): 0.03779 and shape=torch.Size([96, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1305352Z E0601 05:03:41.130000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06321, (ref-fp64): 0.10381 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1309641Z E0601 05:03:41.130000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.17996, (ref-fp64): 0.15558 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1314570Z E0601 05:03:41.131000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03263, (ref-fp64): 0.02906 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1318829Z E0601 05:03:41.131000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07064, (ref-fp64): 0.09528 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1323279Z E0601 05:03:41.131000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.25415, (ref-fp64): 0.22857 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1327952Z E0601 05:03:41.132000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09275, (ref-fp64): 0.07911 and shape=torch.Size([576, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1332153Z E0601 05:03:41.132000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03097, (ref-fp64): 0.04700 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1336383Z E0601 05:03:41.133000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.16921, (ref-fp64): 0.14167 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1340967Z E0601 05:03:41.133000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04592, (ref-fp64): 0.04263 and shape=torch.Size([96, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1345288Z E0601 05:03:41.134000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05834, (ref-fp64): 0.10501 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1349499Z E0601 05:03:41.134000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.35132, (ref-fp64): 0.30436 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1354268Z E0601 05:03:41.135000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06415, (ref-fp64): 0.05746 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1358486Z E0601 05:03:41.135000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06703, (ref-fp64): 0.12337 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1362969Z E0601 05:03:41.135000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.24168, (ref-fp64): 0.28731 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1367503Z E0601 05:03:41.136000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01599, (ref-fp64): 0.02062 and shape=torch.Size([576, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1371923Z E0601 05:03:41.136000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03118, (ref-fp64): 0.06570 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1376106Z E0601 05:03:41.137000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07940, (ref-fp64): 0.08836 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1381208Z E0601 05:03:41.137000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02809, (ref-fp64): 0.03566 and shape=torch.Size([192, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1385548Z E0601 05:03:41.138000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04415, (ref-fp64): 0.08846 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1389758Z E0601 05:03:41.138000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.12699, (ref-fp64): 0.12059 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1394747Z E0601 05:03:41.139000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01966, (ref-fp64): 0.01876 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1399013Z E0601 05:03:41.139000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03746, (ref-fp64): 0.05402 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1403516Z E0601 05:03:41.139000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.11378, (ref-fp64): 0.10640 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1408021Z E0601 05:03:41.140000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02153, (ref-fp64): 0.02065 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1412240Z E0601 05:03:41.140000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01839, (ref-fp64): 0.04424 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1416527Z E0601 05:03:41.141000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07661, (ref-fp64): 0.07790 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1421145Z E0601 05:03:41.141000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02120, (ref-fp64): 0.02270 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1425351Z E0601 05:03:41.142000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03139, (ref-fp64): 0.08237 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1429608Z E0601 05:03:41.142000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.22116, (ref-fp64): 0.21496 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1434547Z E0601 05:03:41.143000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02232, (ref-fp64): 0.02100 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1438915Z E0601 05:03:41.143000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02751, (ref-fp64): 0.07457 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1443351Z E0601 05:03:41.143000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09551, (ref-fp64): 0.10296 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1448105Z E0601 05:03:41.144000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02035, (ref-fp64): 0.02124 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1452255Z E0601 05:03:41.144000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01213, (ref-fp64): 0.03809 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1456412Z E0601 05:03:41.145000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06653, (ref-fp64): 0.06634 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1461004Z E0601 05:03:41.145000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01508, (ref-fp64): 0.01653 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1465269Z E0601 05:03:41.146000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02341, (ref-fp64): 0.06571 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1469531Z E0601 05:03:41.146000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.13304, (ref-fp64): 0.11300 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1474285Z E0601 05:03:41.147000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01785, (ref-fp64): 0.01645 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1478690Z E0601 05:03:41.147000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02129, (ref-fp64): 0.07445 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1482921Z E0601 05:03:41.147000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07170, (ref-fp64): 0.07294 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1487542Z E0601 05:03:41.148000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01480, (ref-fp64): 0.01569 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1491768Z E0601 05:03:41.148000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01249, (ref-fp64): 0.05674 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1495942Z E0601 05:03:41.149000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04913, (ref-fp64): 0.04985 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1500639Z E0601 05:03:41.149000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01548, (ref-fp64): 0.01732 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1504803Z E0601 05:03:41.150000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02066, (ref-fp64): 0.06063 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1509115Z E0601 05:03:41.150000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09983, (ref-fp64): 0.08603 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1513926Z E0601 05:03:41.150000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03389, (ref-fp64): 0.02948 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1518208Z E0601 05:03:41.151000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03256, (ref-fp64): 0.08828 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1522626Z E0601 05:03:41.151000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.12897, (ref-fp64): 0.12435 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1527272Z E0601 05:03:41.152000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03760, (ref-fp64): 0.03642 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1531472Z E0601 05:03:41.152000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03327, (ref-fp64): 0.05029 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1535636Z E0601 05:03:41.153000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.10553, (ref-fp64): 0.08071 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1540373Z E0601 05:03:41.153000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01908, (ref-fp64): 0.02309 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1544760Z E0601 05:03:41.154000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03476, (ref-fp64): 0.04862 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1549052Z E0601 05:03:41.154000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.17153, (ref-fp64): 0.17920 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1554101Z E0601 05:03:41.154000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02250, (ref-fp64): 0.01902 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1559806Z E0601 05:03:41.155000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00393, (ref-fp64): 0.00452 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1564170Z E0601 05:03:41.156000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01444, (ref-fp64): 0.01230 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1568639Z E0601 05:03:41.156000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02709, (ref-fp64): 0.08048 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1572843Z E0601 05:03:41.156000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02378, (ref-fp64): 0.06138 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1577241Z E0601 05:03:41.157000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03562, (ref-fp64): 0.13704 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1581559Z E0601 05:03:41.157000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01920, (ref-fp64): 0.04818 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1585819Z E0601 05:03:41.158000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02890, (ref-fp64): 0.07795 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1590348Z E0601 05:03:41.158000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03299, (ref-fp64): 0.14053 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1594852Z E0601 05:03:41.159000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00945, (ref-fp64): 0.02825 and shape=torch.Size([48, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1599065Z E0601 05:03:41.159000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05578, (ref-fp64): 0.08074 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1603344Z E0601 05:03:41.159000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05982, (ref-fp64): 0.14639 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1607895Z E0601 05:03:41.160000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03237, (ref-fp64): 0.05394 and shape=torch.Size([48, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1612027Z E0601 05:03:41.160000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03224, (ref-fp64): 0.08268 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1616387Z E0601 05:03:41.161000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03689, (ref-fp64): 0.15082 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1620611Z E0601 05:03:41.161000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02539, (ref-fp64): 0.06100 and shape=torch.Size([24, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1625291Z E0601 05:03:41.162000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03203, (ref-fp64): 0.07933 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1629542Z E0601 05:03:41.162000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06596, (ref-fp64): 0.17500 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1634227Z E0601 05:03:41.162000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00860, (ref-fp64): 0.02043 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1638150Z E0601 05:03:41.163000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01699, (ref-fp64): 0.03731 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1642602Z E0601 05:03:41.163000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03462, (ref-fp64): 0.08450 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1647241Z E0601 05:03:41.164000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02517, (ref-fp64): 0.05456 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1651383Z E0601 05:03:41.164000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01889, (ref-fp64): 0.04687 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1655604Z E0601 05:03:41.165000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02112, (ref-fp64): 0.06935 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1660039Z E0601 05:03:41.165000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01899, (ref-fp64): 0.04418 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1664029Z E0601 05:03:41.166000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03356, (ref-fp64): 0.07669 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1668288Z E0601 05:03:41.166000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05551, (ref-fp64): 0.11426 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1672893Z E0601 05:03:41.166000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01008, (ref-fp64): 0.02095 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1677245Z E0601 05:03:41.167000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03227, (ref-fp64): 0.06380 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1681597Z E0601 05:03:41.167000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06033, (ref-fp64): 0.10066 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1686210Z E0601 05:03:41.168000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03917, (ref-fp64): 0.05017 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1690434Z E0601 05:03:41.168000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04140, (ref-fp64): 0.07617 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1694702Z E0601 05:03:41.169000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03715, (ref-fp64): 0.05740 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1699040Z E0601 05:03:41.169000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02147, (ref-fp64): 0.04782 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1703267Z E0601 05:03:41.169000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03730, (ref-fp64): 0.08759 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1707681Z E0601 05:03:41.170000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03282, (ref-fp64): 0.06532 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1712096Z E0601 05:03:41.170000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01603, (ref-fp64): 0.03157 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1716453Z E0601 05:03:41.171000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06388, (ref-fp64): 0.18757 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1720559Z E0601 05:03:41.171000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06585, (ref-fp64): 0.18512 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1725190Z E0601 05:03:41.172000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01451, (ref-fp64): 0.03570 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1729383Z E0601 05:03:41.172000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03601, (ref-fp64): 0.09363 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1733687Z E0601 05:03:41.172000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04657, (ref-fp64): 0.12425 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1738211Z E0601 05:03:41.173000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02384, (ref-fp64): 0.05614 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1742352Z E0601 05:03:41.173000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02991, (ref-fp64): 0.08235 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1746641Z E0601 05:03:41.174000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04291, (ref-fp64): 0.08526 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1751197Z E0601 05:03:41.174000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00951, (ref-fp64): 0.01525 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1755546Z E0601 05:03:41.175000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03477, (ref-fp64): 0.07681 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1759692Z E0601 05:03:41.175000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04559, (ref-fp64): 0.15105 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1764324Z E0601 05:03:41.176000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01990, (ref-fp64): 0.04019 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1768541Z E0601 05:03:41.176000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02226, (ref-fp64): 0.05407 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1772767Z E0601 05:03:41.176000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02249, (ref-fp64): 0.04578 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1777242Z E0601 05:03:41.177000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01147, (ref-fp64): 0.02595 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1781392Z E0601 05:03:41.177000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03058, (ref-fp64): 0.08696 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1785945Z E0601 05:03:41.178000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03302, (ref-fp64): 0.05849 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1790528Z E0601 05:03:41.178000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01297, (ref-fp64): 0.02000 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1794900Z E0601 05:03:41.179000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03987, (ref-fp64): 0.06700 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1799120Z E0601 05:03:41.179000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09803, (ref-fp64): 0.08343 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1803562Z E0601 05:03:41.179000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02797, (ref-fp64): 0.03713 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1807968Z E0601 05:03:41.180000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02631, (ref-fp64): 0.06823 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1812293Z E0601 05:03:41.180000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09533, (ref-fp64): 0.10016 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1816754Z E0601 05:03:41.181000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02868, (ref-fp64): 0.04735 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1820905Z E0601 05:03:41.181000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03173, (ref-fp64): 0.09252 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1825155Z E0601 05:03:41.182000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.08697, (ref-fp64): 0.13766 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1835977Z E0601 05:03:41.183000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00376, (ref-fp64): 0.00390 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1840694Z E0601 05:03:41.183000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01273, (ref-fp64): 0.01398 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1845432Z E0601 05:03:41.184000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01286, (ref-fp64): 0.01136 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1850633Z E0601 05:03:41.184000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01394, (ref-fp64): 0.01277 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1855250Z E0601 05:03:41.185000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01537, (ref-fp64): 0.01865 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1859755Z E0601 05:03:41.185000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01491, (ref-fp64): 0.01896 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1864335Z E0601 05:03:41.186000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01560, (ref-fp64): 0.01871 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1869163Z E0601 05:03:41.186000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01512, (ref-fp64): 0.01833 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1873764Z E0601 05:03:41.186000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01539, (ref-fp64): 0.01775 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1878207Z E0601 05:03:41.187000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01600, (ref-fp64): 0.01771 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1883159Z E0601 05:03:41.187000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01568, (ref-fp64): 0.01861 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1888126Z E0601 05:03:41.188000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01638, (ref-fp64): 0.01871 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1892852Z E0601 05:03:41.188000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01649, (ref-fp64): 0.01762 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1897318Z E0601 05:03:41.189000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01324, (ref-fp64): 0.01535 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1901939Z E0601 05:03:41.189000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01547, (ref-fp64): 0.01664 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1906619Z E0601 05:03:41.190000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01487, (ref-fp64): 0.01674 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1911975Z E0601 05:03:41.190000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01320, (ref-fp64): 0.01566 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1916497Z E0601 05:03:41.191000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01508, (ref-fp64): 0.01744 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1920833Z E0601 05:03:41.191000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01449, (ref-fp64): 0.01699 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1925957Z E0601 05:03:41.192000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01310, (ref-fp64): 0.01537 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1931645Z E0601 05:03:41.192000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01538, (ref-fp64): 0.01779 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1935255Z E0601 05:03:41.193000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01561, (ref-fp64): 0.01740 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1940051Z E0601 05:03:41.193000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01180, (ref-fp64): 0.01425 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1944486Z E0601 05:03:41.194000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01440, (ref-fp64): 0.01538 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1951298Z E0601 05:03:41.194000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01339, (ref-fp64): 0.01495 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1955845Z E0601 05:03:41.195000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01154, (ref-fp64): 0.01439 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.1960187Z E0601 05:03:41.195000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01323, (ref-fp64): 0.01570 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1964948Z E0601 05:03:41.196000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01319, (ref-fp64): 0.01491 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1969933Z E0601 05:03:41.196000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01189, (ref-fp64): 0.01439 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1974476Z E0601 05:03:41.197000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01535, (ref-fp64): 0.01848 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1979003Z E0601 05:03:41.197000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01524, (ref-fp64): 0.01799 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1983829Z E0601 05:03:41.197000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01789 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1988449Z E0601 05:03:41.198000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01585, (ref-fp64): 0.01887 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1993231Z E0601 05:03:41.198000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01559, (ref-fp64): 0.01883 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.1998054Z E0601 05:03:41.199000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01435, (ref-fp64): 0.01755 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2002750Z E0601 05:03:41.199000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01559, (ref-fp64): 0.01789 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2007490Z E0601 05:03:41.200000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01480, (ref-fp64): 0.01800 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2012341Z E0601 05:03:41.200000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01345, (ref-fp64): 0.01691 and shape=torch.Size([96, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2016827Z E0601 05:03:41.201000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01394, (ref-fp64): 0.01720 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2021519Z E0601 05:03:41.201000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01448, (ref-fp64): 0.01741 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2026467Z E0601 05:03:41.202000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01140, (ref-fp64): 0.01373 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2031186Z E0601 05:03:41.202000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01257, (ref-fp64): 0.01473 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2036025Z E0601 05:03:41.203000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01239, (ref-fp64): 0.01478 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2040653Z E0601 05:03:41.203000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01197, (ref-fp64): 0.01382 and shape=torch.Size([576, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2045608Z E0601 05:03:41.204000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01268, (ref-fp64): 0.01483 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2050055Z E0601 05:03:41.204000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01204, (ref-fp64): 0.01401 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2054816Z E0601 05:03:41.205000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01186, (ref-fp64): 0.01418 and shape=torch.Size([96, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2059482Z E0601 05:03:41.205000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01649, (ref-fp64): 0.01881 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2064033Z E0601 05:03:41.206000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01668, (ref-fp64): 0.01761 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2069051Z E0601 05:03:41.206000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01320, (ref-fp64): 0.01706 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2073813Z E0601 05:03:41.206000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01633, (ref-fp64): 0.01899 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2078405Z E0601 05:03:41.207000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01597, (ref-fp64): 0.01921 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2083097Z E0601 05:03:41.207000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01195, (ref-fp64): 0.01524 and shape=torch.Size([576, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2088022Z E0601 05:03:41.208000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01831 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2092519Z E0601 05:03:41.208000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01358, (ref-fp64): 0.01736 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2097778Z E0601 05:03:41.209000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01285, (ref-fp64): 0.01609 and shape=torch.Size([192, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2102359Z E0601 05:03:41.209000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01518, (ref-fp64): 0.01764 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2107009Z E0601 05:03:41.210000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01421, (ref-fp64): 0.01679 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2112405Z E0601 05:03:41.210000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01095, (ref-fp64): 0.01303 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2117165Z E0601 05:03:41.211000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01269, (ref-fp64): 0.01469 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2121527Z E0601 05:03:41.211000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01241, (ref-fp64): 0.01439 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2126740Z E0601 05:03:41.212000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01098, (ref-fp64): 0.01317 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2131296Z E0601 05:03:41.212000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01300, (ref-fp64): 0.01565 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2135927Z E0601 05:03:41.213000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01258, (ref-fp64): 0.01477 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2141052Z E0601 05:03:41.213000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01102, (ref-fp64): 0.01417 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2145670Z E0601 05:03:41.214000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01406, (ref-fp64): 0.01818 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2150227Z E0601 05:03:41.214000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01490, (ref-fp64): 0.01705 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2155372Z E0601 05:03:41.215000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00830, (ref-fp64): 0.01099 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2159562Z E0601 05:03:41.215000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00920, (ref-fp64): 0.01205 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2164512Z E0601 05:03:41.216000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00924, (ref-fp64): 0.01194 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2169252Z E0601 05:03:41.216000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00788, (ref-fp64): 0.01061 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2174025Z E0601 05:03:41.216000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00994, (ref-fp64): 0.01261 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2178573Z E0601 05:03:41.217000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00936, (ref-fp64): 0.01187 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2183533Z E0601 05:03:41.217000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00956, (ref-fp64): 0.01172 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2188142Z E0601 05:03:41.218000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01769 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2193411Z E0601 05:03:41.218000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01554, (ref-fp64): 0.01810 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2198050Z E0601 05:03:41.219000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00801, (ref-fp64): 0.01068 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2202648Z E0601 05:03:41.219000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00933, (ref-fp64): 0.01148 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2207598Z E0601 05:03:41.220000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00950, (ref-fp64): 0.01153 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2212529Z E0601 05:03:41.220000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00768, (ref-fp64): 0.01008 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2217635Z E0601 05:03:41.221000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00994, (ref-fp64): 0.01206 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2222409Z E0601 05:03:41.221000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00932, (ref-fp64): 0.01108 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2227479Z E0601 05:03:41.222000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00957, (ref-fp64): 0.01130 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2232252Z E0601 05:03:41.222000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01479, (ref-fp64): 0.01733 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2237135Z E0601 05:03:41.223000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01397, (ref-fp64): 0.01666 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2242359Z E0601 05:03:41.223000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00889, (ref-fp64): 0.01192 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2247207Z E0601 05:03:41.224000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01211, (ref-fp64): 0.01371 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2251924Z E0601 05:03:41.224000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01209, (ref-fp64): 0.01373 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2256566Z E0601 05:03:41.225000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01010, (ref-fp64): 0.01189 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2261506Z E0601 05:03:41.225000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01254, (ref-fp64): 0.01353 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2266298Z E0601 05:03:41.226000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01176, (ref-fp64): 0.01245 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2271537Z E0601 05:03:41.226000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01275, (ref-fp64): 0.01334 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2276322Z E0601 05:03:41.227000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01880, (ref-fp64): 0.01861 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2281164Z E0601 05:03:41.227000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01323, (ref-fp64): 0.01518 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2286587Z E0601 05:03:41.228000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00989, (ref-fp64): 0.01316 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2291170Z E0601 05:03:41.228000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01026, (ref-fp64): 0.01252 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2295856Z E0601 05:03:41.229000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01012, (ref-fp64): 0.01235 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2300743Z E0601 05:03:41.229000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01059, (ref-fp64): 0.01216 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2305679Z E0601 05:03:41.230000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01157, (ref-fp64): 0.01323 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2319044Z E0601 05:03:41.230000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01269, (ref-fp64): 0.01405 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2321882Z E0601 05:03:41.231000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01094, (ref-fp64): 0.01377 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2324283Z E0601 05:03:41.231000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01136, (ref-fp64): 0.01806 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2326380Z E0601 05:03:41.232000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01732, (ref-fp64): 0.01719 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2330249Z E0601 05:03:41.232000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01541, (ref-fp64): 0.01753 and shape=torch.Size([48, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2334582Z E0601 05:03:41.233000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01255, (ref-fp64): 0.01760 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2339430Z E0601 05:03:41.233000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01557, (ref-fp64): 0.01710 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2344311Z E0601 05:03:41.234000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01727 and shape=torch.Size([48, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2349099Z E0601 05:03:41.234000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01320, (ref-fp64): 0.01806 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2353877Z E0601 05:03:41.234000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01531, (ref-fp64): 0.01833 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2358775Z E0601 05:03:41.235000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01412, (ref-fp64): 0.01888 and shape=torch.Size([24, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2363613Z E0601 05:03:41.235000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01331, (ref-fp64): 0.02003 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2368582Z E0601 05:03:41.236000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02139, (ref-fp64): 0.02007 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2373479Z E0601 05:03:41.236000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01564, (ref-fp64): 0.01744 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2378095Z E0601 05:03:41.237000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01553, (ref-fp64): 0.01865 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2382962Z E0601 05:03:41.237000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01691, (ref-fp64): 0.01816 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2387741Z E0601 05:03:41.238000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01583, (ref-fp64): 0.01706 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2392823Z E0601 05:03:41.238000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01540, (ref-fp64): 0.01855 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2397603Z E0601 05:03:41.239000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01560, (ref-fp64): 0.01818 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2402660Z E0601 05:03:41.239000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01609, (ref-fp64): 0.01825 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2407452Z E0601 05:03:41.240000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01564, (ref-fp64): 0.01894 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2412216Z E0601 05:03:41.240000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01736, (ref-fp64): 0.01869 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2417138Z E0601 05:03:41.241000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01480, (ref-fp64): 0.01734 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2421742Z E0601 05:03:41.241000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01549, (ref-fp64): 0.01762 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2426560Z E0601 05:03:41.242000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01573, (ref-fp64): 0.01765 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2431623Z E0601 05:03:41.242000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01585, (ref-fp64): 0.01694 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2436334Z E0601 05:03:41.243000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01688, (ref-fp64): 0.01765 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2441062Z E0601 05:03:41.243000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01663, (ref-fp64): 0.01845 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2446263Z E0601 05:03:41.244000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01658, (ref-fp64): 0.01882 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2450977Z E0601 05:03:41.244000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01805, (ref-fp64): 0.02010 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2455585Z E0601 05:03:41.245000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01404, (ref-fp64): 0.01566 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2460540Z E0601 05:03:41.245000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01404, (ref-fp64): 0.01726 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2465276Z E0601 05:03:41.246000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01769, (ref-fp64): 0.01673 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2470333Z E0601 05:03:41.246000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01796, (ref-fp64): 0.01740 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2475426Z E0601 05:03:41.247000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01768, (ref-fp64): 0.01665 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2479869Z E0601 05:03:41.247000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01838, (ref-fp64): 0.01688 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2484650Z E0601 05:03:41.248000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01499, (ref-fp64): 0.01622 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2489649Z E0601 05:03:41.248000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01650, (ref-fp64): 0.01701 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2494436Z E0601 05:03:41.249000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01634, (ref-fp64): 0.01735 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2499102Z E0601 05:03:41.249000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01675, (ref-fp64): 0.01667 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2503985Z E0601 05:03:41.249000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01618 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2508689Z E0601 05:03:41.250000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01365, (ref-fp64): 0.01522 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2513608Z E0601 05:03:41.250000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01737, (ref-fp64): 0.01668 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2518426Z E0601 05:03:41.251000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01642, (ref-fp64): 0.01613 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2523263Z E0601 05:03:41.251000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01716, (ref-fp64): 0.01565 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2528243Z E0601 05:03:41.252000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01737, (ref-fp64): 0.01668 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2533156Z E0601 05:03:41.252000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01636 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2537947Z E0601 05:03:41.253000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01465, (ref-fp64): 0.01702 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2542598Z E0601 05:03:41.253000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01770, (ref-fp64): 0.01590 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2548215Z E0601 05:03:41.254000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01413, (ref-fp64): 0.01689 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2552619Z E0601 05:03:41.254000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01505, (ref-fp64): 0.01749 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2557041Z E0601 05:03:41.255000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01470, (ref-fp64): 0.01716 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2562035Z E0601 05:03:41.255000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01533, (ref-fp64): 0.01643 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:03:41.2567092Z E0601 05:03:41.256000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01591, (ref-fp64): 0.01651 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2571639Z E0601 05:03:41.256000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01576, (ref-fp64): 0.01694 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2576619Z E0601 05:03:41.257000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01462, (ref-fp64): 0.01599 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2581262Z E0601 05:03:41.257000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01657, (ref-fp64): 0.01697 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2585902Z E0601 05:03:41.258000 139711399002752 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01798, (ref-fp64): 0.01800 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:03:41.2932352Z pass 2024-06-01T05:03:41.2960987Z TIMING: entire_frame_compile:81.841 code_gen:26.48098 inductor_compile:47.6926 backend_compile:71.41664 2024-06-01T05:03:41.2962623Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:14137 | FakeTensorMode.__torch_dispatch__:86214 | attempt fast:1489 | fast is_contiguous:1489 | ProxyTorchDispatchMode.__torch_dispatch__:17678 2024-06-01T05:03:41.2964032Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-06-01T05:03:48.5220180Z 2024-06-01T05:03:50.6921972Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:03:50.6922596Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:03:50.6923825Z cuda train mobilenet_v2 2024-06-01T05:04:45.0730072Z E0601 05:04:45.072000 140669288944256 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00548, (ref-fp64): 0.00539 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:04:45.0890629Z E0601 05:04:45.088000 140669288944256 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00001, (ref-fp64): 0.00007 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:04:45.1706522Z pass 2024-06-01T05:04:45.1755725Z TIMING: entire_frame_compile:49.11385 code_gen:10.32241 inductor_compile:26.87967 backend_compile:43.23176 2024-06-01T05:04:45.1757674Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:8000 | FakeTensorMode.__torch_dispatch__:52312 | attempt fast:1661 | fast is_contiguous:1661 | ProxyTorchDispatchMode.__torch_dispatch__:10584 2024-06-01T05:04:45.1759103Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-06-01T05:04:51.1017584Z 2024-06-01T05:04:53.0456195Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:04:53.0456771Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:04:53.0457295Z cuda train mobilenet_v2_quantized_qat 2024-06-01T05:04:53.0457793Z Traceback (most recent call last): 2024-06-01T05:04:53.0458538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model 2024-06-01T05:04:53.0459326Z self.model_iter_fn(model, example_inputs) 2024-06-01T05:04:53.0460437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in forward_and_backward_pass 2024-06-01T05:04:53.0461295Z pred = mod(*cloned_inputs) 2024-06-01T05:04:53.0462302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 737, in call_wrapped 2024-06-01T05:04:53.0463245Z return self._wrapped_call(self, *args, **kwargs) 2024-06-01T05:04:53.0464207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__ 2024-06-01T05:04:53.0465267Z raise e 2024-06-01T05:04:53.0466306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__ 2024-06-01T05:04:53.0467509Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-01T05:04:53.0468792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1540, in _wrapped_call_impl 2024-06-01T05:04:53.0469720Z return self._call_impl(*args, **kwargs) 2024-06-01T05:04:53.0471127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:04:53.0472099Z return forward_call(*args, **kwargs) 2024-06-01T05:04:53.0472583Z File ".3", line 207, in forward 2024-06-01T05:04:53.0473360Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-06-01T05:04:53.0474635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1540, in _wrapped_call_impl 2024-06-01T05:04:53.0475543Z return self._call_impl(*args, **kwargs) 2024-06-01T05:04:53.0476485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:04:53.0477336Z return forward_call(*args, **kwargs) 2024-06-01T05:04:53.0478322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward 2024-06-01T05:04:53.0479271Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-01T05:04:53.0479851Z RuntimeError: expected scalar type Float but found Half 2024-06-01T05:04:53.0480241Z 2024-06-01T05:04:53.0480555Z The above exception was the direct cause of the following exception: 2024-06-01T05:04:53.0481298Z 2024-06-01T05:04:53.0481461Z Traceback (most recent call last): 2024-06-01T05:04:53.0482277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T05:04:53.0482960Z ) = runner.load_model( 2024-06-01T05:04:53.0483647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model 2024-06-01T05:04:53.0484437Z self.validate_model(model, example_inputs) 2024-06-01T05:04:53.0485216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model 2024-06-01T05:04:53.0486010Z raise RuntimeError("Eager run failed") from e 2024-06-01T05:04:53.0486499Z RuntimeError: Eager run failed 2024-06-01T05:04:53.0486765Z 2024-06-01T05:04:53.0486890Z eager_fail_to_run 2024-06-01T05:04:56.3434703Z 2024-06-01T05:04:59.0015574Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:04:59.0016055Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:04:59.0016511Z cuda train mobilenet_v3_large 2024-06-01T05:06:00.0193844Z W0601 05:06:00.018000 140690841821824 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:06:47.9948794Z E0601 05:06:47.993000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5.07696, (ref-fp64): 10.21581 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:47.9956684Z E0601 05:06:47.995000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.89639, (ref-fp64): 5.43459 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:47.9960657Z E0601 05:06:47.995000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00059, (ref-fp64): 0.00119 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:47.9968208Z E0601 05:06:47.996000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01166, (ref-fp64): 0.02418 and shape=torch.Size([1280, 960]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:47.9978622Z E0601 05:06:47.997000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00225 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:47.9983255Z E0601 05:06:47.997000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.18701, (ref-fp64): 0.28992 and shape=torch.Size([16, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:47.9987927Z E0601 05:06:47.998000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.63914, (ref-fp64): 2.99988 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:47.9993863Z E0601 05:06:47.998000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.98862, (ref-fp64): 2.12358 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:47.9998135Z E0601 05:06:47.999000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.59090, (ref-fp64): 3.32086 and shape=torch.Size([16, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0002862Z E0601 05:06:47.999000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.33764, (ref-fp64): 2.75850 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0007821Z E0601 05:06:48.000000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.33817, (ref-fp64): 2.49605 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0012667Z E0601 05:06:48.000000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.06994, (ref-fp64): 2.03366 and shape=torch.Size([16, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0017244Z E0601 05:06:48.001000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.03175, (ref-fp64): 2.18528 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0021979Z E0601 05:06:48.001000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.39050, (ref-fp64): 2.72181 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0026885Z E0601 05:06:48.002000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.17285, (ref-fp64): 1.38729 and shape=torch.Size([184, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0031816Z E0601 05:06:48.002000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.26247, (ref-fp64): 0.39907 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0036600Z E0601 05:06:48.003000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8.85797, (ref-fp64): 12.83275 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0041399Z E0601 05:06:48.003000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2.36670, (ref-fp64): 3.42576 and shape=torch.Size([184, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:48.0046442Z E0601 05:06:48.004000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.08719, (ref-fp64): 0.11251 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0051051Z E0601 05:06:48.004000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2.40322, (ref-fp64): 3.56040 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0055884Z E0601 05:06:48.005000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.43831, (ref-fp64): 1.53879 and shape=torch.Size([80, 184, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0060636Z E0601 05:06:48.005000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06100, (ref-fp64): 0.05932 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0065177Z E0601 05:06:48.006000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 6.73719, (ref-fp64): 6.16697 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0070660Z E0601 05:06:48.006000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.83392, (ref-fp64): 0.73169 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0075260Z E0601 05:06:48.007000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.17559, (ref-fp64): 0.13162 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0079927Z E0601 05:06:48.007000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3.58612, (ref-fp64): 4.69831 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0084918Z E0601 05:06:48.008000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.81642, (ref-fp64): 1.04145 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:48.0089767Z E0601 05:06:48.008000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06072, (ref-fp64): 0.04568 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0094422Z E0601 05:06:48.009000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.69280, (ref-fp64): 0.91381 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0099301Z E0601 05:06:48.009000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00754, (ref-fp64): 0.00901 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0104285Z E0601 05:06:48.010000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.17653, (ref-fp64): 0.21214 and shape=torch.Size([120, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0109236Z E0601 05:06:48.010000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00917, (ref-fp64): 0.00986 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0114318Z E0601 05:06:48.011000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.46603, (ref-fp64): 0.50059 and shape=torch.Size([480, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0119320Z E0601 05:06:48.011000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.01267, (ref-fp64): 0.68357 and shape=torch.Size([112, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0124160Z E0601 05:06:48.012000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.11619, (ref-fp64): 0.07863 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0128780Z E0601 05:06:48.012000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4.69317, (ref-fp64): 2.59594 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0134167Z E0601 05:06:48.013000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.22387, (ref-fp64): 0.81187 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0138839Z E0601 05:06:48.013000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07796, (ref-fp64): 0.05858 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0143522Z E0601 05:06:48.013000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2.55956, (ref-fp64): 1.76422 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0148545Z E0601 05:06:48.014000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.37128, (ref-fp64): 1.05518 and shape=torch.Size([672, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:48.0153141Z E0601 05:06:48.014000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02413, (ref-fp64): 0.01925 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0158036Z E0601 05:06:48.015000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.23482, (ref-fp64): 0.85447 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0162728Z E0601 05:06:48.015000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02342, (ref-fp64): 0.01271 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0168716Z E0601 05:06:48.016000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.67198, (ref-fp64): 0.37847 and shape=torch.Size([168, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0173323Z E0601 05:06:48.016000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02788, (ref-fp64): 0.01440 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0178974Z E0601 05:06:48.017000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.15461, (ref-fp64): 0.63004 and shape=torch.Size([672, 168, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0184368Z E0601 05:06:48.018000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.19965, (ref-fp64): 0.32251 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0189065Z E0601 05:06:48.018000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02402, (ref-fp64): 0.02346 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0193870Z E0601 05:06:48.018000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.04451, (ref-fp64): 1.78398 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0200152Z E0601 05:06:48.019000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.34770, (ref-fp64): 0.76521 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0204053Z E0601 05:06:48.019000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04576, (ref-fp64): 0.05317 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0209183Z E0601 05:06:48.020000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.69187, (ref-fp64): 4.21924 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0213928Z E0601 05:06:48.020000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.36779, (ref-fp64): 3.08061 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:06:48.0219549Z E0601 05:06:48.021000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06060, (ref-fp64): 0.09749 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0223458Z E0601 05:06:48.021000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.52387, (ref-fp64): 1.39598 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0228256Z E0601 05:06:48.022000 140690841821824 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00192, (ref-fp64): 0.00024 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:06:48.0231565Z E0601 05:06:48.022000 140690841821824 torch/_dynamo/utils.py:1314] Accuracy failed for key name features.13.block.2.fc1.bias.grad 2024-06-01T05:06:48.0261462Z pass 2024-06-01T05:06:48.0299440Z TIMING: entire_frame_compile:101.00838 code_gen:27.20573 inductor_compile:56.39696 backend_compile:86.79719 2024-06-01T05:06:48.0301029Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:16968 | FakeTensorMode.__torch_dispatch__:102336 | attempt fast:2123 | fast is_contiguous:2123 | ProxyTorchDispatchMode.__torch_dispatch__:19585 2024-06-01T05:06:48.0302406Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-06-01T05:06:56.3882284Z 2024-06-01T05:06:58.6792677Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:06:58.6793403Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:06:58.6794077Z cuda train moco 2024-06-01T05:07:04.7751110Z [rank0]:W0601 05:07:04.774000 139744211202688 torch/_logging/_internal.py:1033] [3/0] Profiler function will be ignored 2024-06-01T05:07:12.3005846Z [rank0]:W0601 05:07:12.299000 139744211202688 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-01T05:07:12.3196168Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-01T05:07:12.3202547Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-01T05:07:12.3204331Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-01T05:07:12.3205899Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-01T05:07:12.3207420Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-06-01T05:07:12.3208968Z [rank0]:W0601 05:07:12.318000 139744211202688 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-01T05:08:04.0250757Z skipping cudagraphs due to mutated inputs (161 instances) 2024-06-01T05:08:06.2369162Z [rank0]:W0601 05:08:06.236000 139744211202688 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:08:06.2552755Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-06-01T05:08:06.2554641Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-06-01T05:08:06.2555679Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-06-01T05:08:06.2556055Z 2024-06-01T05:08:08.2948953Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-06-01T05:08:08.2950552Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-01T05:08:08.2951741Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] or: 2024-06-01T05:08:08.2952875Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-01T05:08:08.2954270Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] to include these operations in the captured graph. 2024-06-01T05:08:08.2955430Z [rank0]:W0601 05:08:08.294000 139744211202688 torch/_dynamo/variables/tensor.py:715] [7/0] 2024-06-01T05:08:36.2505631Z [rank0]:E0601 05:08:36.249000 139744211202688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01932, (ref-fp64): 0.01936 and shape=torch.Size([4, 32001]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:08:36.4553050Z pass 2024-06-01T05:08:36.4887153Z TIMING: entire_frame_compile:83.58392 inductor_compile:42.31415 backend_compile:71.60094 code_gen:19.05415 2024-06-01T05:08:36.4888873Z STATS: call_* op count: 907 | FakeTensorMode.__torch_dispatch__:94161 | ProxyTorchDispatchMode.__torch_dispatch__:20700 | FakeTensor.__torch_dispatch__:13451 | attempt fast:2739 | fast is_contiguous:2723 | slow no contiguity match:16 2024-06-01T05:08:36.4891905Z Dynamo produced 10 graphs covering 907 ops with 11 graph breaks (8 unique) 2024-06-01T05:08:43.4778684Z 2024-06-01T05:08:45.3628826Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-01T05:08:45.5794889Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-01T05:08:45.5795777Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-01T05:08:45.5798727Z using fused AdamW: True 2024-06-01T05:08:45.9634233Z 2024-06-01T05:08:45.9634774Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:08:45.9635275Z cuda train nanogpt 2024-06-01T05:09:38.4936065Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:09:38.4939421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in torch_dynamo_resume_in_forward_and_backward_pass_at_434 2024-06-01T05:09:38.4940685Z pred = mod(*cloned_inputs) 2024-06-01T05:09:38.4941900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:09:38.4943037Z return forward_call(*args, **kwargs) 2024-06-01T05:09:38.4943913Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward 2024-06-01T05:09:38.4944980Z tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) 2024-06-01T05:09:38.4945494Z 2024-06-01T05:09:39.8233272Z W0601 05:09:39.822000 139635068625536 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:10:17.0568906Z E0601 05:10:17.056000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03315, (ref-fp64): 0.10627 and shape=torch.Size([4, 1, 50304]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0573765Z E0601 05:10:17.056000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00776, (ref-fp64): 0.05881 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.0586791Z E0601 05:10:17.058000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0590781Z E0601 05:10:17.058000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00021, (ref-fp64): 0.00035 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.0595737Z E0601 05:10:17.059000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00016, (ref-fp64): 0.00027 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0606029Z E0601 05:10:17.060000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00010, (ref-fp64): 0.00016 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.0622631Z E0601 05:10:17.061000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00005, (ref-fp64): 0.00008 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0636930Z E0601 05:10:17.063000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00007, (ref-fp64): 0.00010 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0706226Z E0601 05:10:17.070000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00004, (ref-fp64): 0.00006 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0726508Z E0601 05:10:17.072000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00004 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0754439Z E0601 05:10:17.075000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0776294Z E0601 05:10:17.077000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00013 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0815158Z E0601 05:10:17.081000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00014 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0852759Z E0601 05:10:17.084000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00009, (ref-fp64): 0.00018 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0883740Z E0601 05:10:17.087000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0897299Z E0601 05:10:17.089000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00016 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0927993Z E0601 05:10:17.092000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00010 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0939685Z E0601 05:10:17.093000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00006 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0946441Z E0601 05:10:17.094000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00014 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0968916Z E0601 05:10:17.096000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0983543Z E0601 05:10:17.097000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.0995399Z E0601 05:10:17.099000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00004, (ref-fp64): 0.00006 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1022293Z E0601 05:10:17.101000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1036632Z E0601 05:10:17.103000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00007, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1070396Z E0601 05:10:17.106000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1084630Z E0601 05:10:17.108000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1126674Z E0601 05:10:17.112000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00014 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1135269Z E0601 05:10:17.112000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00058, (ref-fp64): 0.00158 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1139470Z E0601 05:10:17.113000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([1024, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1306641Z E0601 05:10:17.130000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00001, (ref-fp64): 0.00001 and shape=torch.Size([50304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1311700Z E0601 05:10:17.130000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00885, (ref-fp64): 0.00904 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1321538Z E0601 05:10:17.131000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00533, (ref-fp64): 0.00621 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1326247Z E0601 05:10:17.132000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00205, (ref-fp64): 0.00379 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1331063Z E0601 05:10:17.132000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00332, (ref-fp64): 0.00476 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1335443Z E0601 05:10:17.133000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00227, (ref-fp64): 0.00385 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1340048Z E0601 05:10:17.133000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00482, (ref-fp64): 0.00625 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1344645Z E0601 05:10:17.134000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00286, (ref-fp64): 0.00398 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1348881Z E0601 05:10:17.134000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00369, (ref-fp64): 0.00517 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1353644Z E0601 05:10:17.134000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00275, (ref-fp64): 0.00443 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1366419Z E0601 05:10:17.136000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00396, (ref-fp64): 0.00511 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1370751Z E0601 05:10:17.136000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00213, (ref-fp64): 0.00420 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1383359Z E0601 05:10:17.137000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00313, (ref-fp64): 0.00420 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1387807Z E0601 05:10:17.138000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00915, (ref-fp64): 0.00925 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1397614Z E0601 05:10:17.139000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00596, (ref-fp64): 0.00664 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1402237Z E0601 05:10:17.139000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00267, (ref-fp64): 0.00365 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1407489Z E0601 05:10:17.140000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00284, (ref-fp64): 0.00417 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1411707Z E0601 05:10:17.140000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00344, (ref-fp64): 0.00468 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1416080Z E0601 05:10:17.141000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00463, (ref-fp64): 0.00586 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1420470Z E0601 05:10:17.141000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00334, (ref-fp64): 0.00473 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1424901Z E0601 05:10:17.142000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00398, (ref-fp64): 0.00506 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1429417Z E0601 05:10:17.142000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00310, (ref-fp64): 0.00458 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1442022Z E0601 05:10:17.143000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00394, (ref-fp64): 0.00528 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1446803Z E0601 05:10:17.144000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00266, (ref-fp64): 0.00397 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1459217Z E0601 05:10:17.145000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00295, (ref-fp64): 0.00408 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1463626Z E0601 05:10:17.145000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00885, (ref-fp64): 0.00894 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1473721Z E0601 05:10:17.146000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00858, (ref-fp64): 0.00882 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1478166Z E0601 05:10:17.147000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00199, (ref-fp64): 0.00262 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1483161Z E0601 05:10:17.147000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00261, (ref-fp64): 0.00407 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1487921Z E0601 05:10:17.148000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00354, (ref-fp64): 0.00491 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1492308Z E0601 05:10:17.148000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00443, (ref-fp64): 0.00597 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1496719Z E0601 05:10:17.149000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00435, (ref-fp64): 0.00761 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1501065Z E0601 05:10:17.149000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00532, (ref-fp64): 0.00852 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1505580Z E0601 05:10:17.150000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00414, (ref-fp64): 0.00697 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1518121Z E0601 05:10:17.151000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00448, (ref-fp64): 0.00707 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1522610Z E0601 05:10:17.151000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00024, (ref-fp64): 0.00103 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1535301Z E0601 05:10:17.153000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00243, (ref-fp64): 0.00353 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1539562Z E0601 05:10:17.153000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00814, (ref-fp64): 0.00819 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1549402Z E0601 05:10:17.154000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00771, (ref-fp64): 0.00785 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1556668Z E0601 05:10:17.155000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00223, (ref-fp64): 0.00307 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1561281Z E0601 05:10:17.155000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00307, (ref-fp64): 0.00309 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1566020Z E0601 05:10:17.156000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00347, (ref-fp64): 0.00428 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1570239Z E0601 05:10:17.156000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00187, (ref-fp64): 0.00322 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1574499Z E0601 05:10:17.157000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00255, (ref-fp64): 0.00396 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1579539Z E0601 05:10:17.157000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00288, (ref-fp64): 0.00285 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1591974Z E0601 05:10:17.158000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00324, (ref-fp64): 0.00372 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1606332Z E0601 05:10:17.160000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00169, (ref-fp64): 0.00278 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1610927Z E0601 05:10:17.160000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01029, (ref-fp64): 0.01067 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1621000Z E0601 05:10:17.161000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00816 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1625444Z E0601 05:10:17.162000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00222, (ref-fp64): 0.00358 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1630595Z E0601 05:10:17.162000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00294, (ref-fp64): 0.00439 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1634970Z E0601 05:10:17.163000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00283, (ref-fp64): 0.00459 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1639377Z E0601 05:10:17.163000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00377, (ref-fp64): 0.00481 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1644186Z E0601 05:10:17.163000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00368, (ref-fp64): 0.00522 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1648547Z E0601 05:10:17.164000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00402, (ref-fp64): 0.00557 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1653012Z E0601 05:10:17.164000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00394, (ref-fp64): 0.00544 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1665535Z E0601 05:10:17.166000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00447, (ref-fp64): 0.00573 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1670527Z E0601 05:10:17.166000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00154, (ref-fp64): 0.00275 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1682679Z E0601 05:10:17.167000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00289, (ref-fp64): 0.00414 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1687709Z E0601 05:10:17.168000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01143, (ref-fp64): 0.01183 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1697636Z E0601 05:10:17.169000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00889, (ref-fp64): 0.00930 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1702278Z E0601 05:10:17.169000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00213, (ref-fp64): 0.00310 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1706745Z E0601 05:10:17.170000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00295, (ref-fp64): 0.00432 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1711256Z E0601 05:10:17.170000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00425, (ref-fp64): 0.00579 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1715870Z E0601 05:10:17.171000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00423, (ref-fp64): 0.00602 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1720099Z E0601 05:10:17.171000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00337, (ref-fp64): 0.00523 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1724790Z E0601 05:10:17.172000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00439, (ref-fp64): 0.00529 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1729639Z E0601 05:10:17.172000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00311, (ref-fp64): 0.00484 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1742190Z E0601 05:10:17.173000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00377, (ref-fp64): 0.00509 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1746883Z E0601 05:10:17.174000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00168, (ref-fp64): 0.00282 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1760875Z E0601 05:10:17.175000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00288, (ref-fp64): 0.00415 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1765850Z E0601 05:10:17.176000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01105, (ref-fp64): 0.01111 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1775716Z E0601 05:10:17.177000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00930, (ref-fp64): 0.00967 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1780463Z E0601 05:10:17.177000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00168, (ref-fp64): 0.00356 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1785413Z E0601 05:10:17.178000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00305, (ref-fp64): 0.00471 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1790266Z E0601 05:10:17.178000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00328, (ref-fp64): 0.00509 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1795036Z E0601 05:10:17.179000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00388, (ref-fp64): 0.00555 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1799943Z E0601 05:10:17.179000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00318, (ref-fp64): 0.00533 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1804649Z E0601 05:10:17.180000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00361, (ref-fp64): 0.00571 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1809293Z E0601 05:10:17.180000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00317, (ref-fp64): 0.00526 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1822190Z E0601 05:10:17.181000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00375, (ref-fp64): 0.00542 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1826963Z E0601 05:10:17.182000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00200, (ref-fp64): 0.00359 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1839274Z E0601 05:10:17.183000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00294, (ref-fp64): 0.00430 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1844258Z E0601 05:10:17.184000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01125, (ref-fp64): 0.01149 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1854335Z E0601 05:10:17.185000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00963, (ref-fp64): 0.00991 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1859420Z E0601 05:10:17.185000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00223, (ref-fp64): 0.00385 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1864131Z E0601 05:10:17.186000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00277, (ref-fp64): 0.00472 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1868808Z E0601 05:10:17.186000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00342, (ref-fp64): 0.00474 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1873706Z E0601 05:10:17.186000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00443, (ref-fp64): 0.00572 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1878495Z E0601 05:10:17.187000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00358, (ref-fp64): 0.00519 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1883122Z E0601 05:10:17.187000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00361, (ref-fp64): 0.00633 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1888322Z E0601 05:10:17.188000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00334, (ref-fp64): 0.00485 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1900867Z E0601 05:10:17.189000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00377, (ref-fp64): 0.00557 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1905490Z E0601 05:10:17.190000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00233, (ref-fp64): 0.00376 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1918212Z E0601 05:10:17.191000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00287, (ref-fp64): 0.00421 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1922562Z E0601 05:10:17.191000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01016, (ref-fp64): 0.01042 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1932785Z E0601 05:10:17.192000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00937, (ref-fp64): 0.00964 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1937213Z E0601 05:10:17.193000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00235, (ref-fp64): 0.00340 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1942131Z E0601 05:10:17.193000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00280, (ref-fp64): 0.00429 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1946526Z E0601 05:10:17.194000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00401, (ref-fp64): 0.00572 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1951144Z E0601 05:10:17.194000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00452, (ref-fp64): 0.00658 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1955723Z E0601 05:10:17.195000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00282, (ref-fp64): 0.00502 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1959922Z E0601 05:10:17.195000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00407, (ref-fp64): 0.00619 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1964848Z E0601 05:10:17.196000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00414, (ref-fp64): 0.00624 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1977329Z E0601 05:10:17.197000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00450, (ref-fp64): 0.00657 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1981737Z E0601 05:10:17.197000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00170, (ref-fp64): 0.00359 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.1993928Z E0601 05:10:17.198000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00266, (ref-fp64): 0.00441 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.1998357Z E0601 05:10:17.199000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00978, (ref-fp64): 0.01001 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2008683Z E0601 05:10:17.200000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00910, (ref-fp64): 0.00937 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2013058Z E0601 05:10:17.200000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00243, (ref-fp64): 0.00375 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2017930Z E0601 05:10:17.201000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00296, (ref-fp64): 0.00449 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2022341Z E0601 05:10:17.201000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00314, (ref-fp64): 0.00510 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2026729Z E0601 05:10:17.202000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00405, (ref-fp64): 0.00637 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2031326Z E0601 05:10:17.202000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00426, (ref-fp64): 0.00676 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2035696Z E0601 05:10:17.203000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00378, (ref-fp64): 0.00668 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2040012Z E0601 05:10:17.203000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00412, (ref-fp64): 0.00688 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2052685Z E0601 05:10:17.204000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00453, (ref-fp64): 0.00703 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2057043Z E0601 05:10:17.205000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00192, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2069578Z E0601 05:10:17.206000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00276, (ref-fp64): 0.00425 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2074428Z E0601 05:10:17.207000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00969, (ref-fp64): 0.01000 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2084575Z E0601 05:10:17.208000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00918, (ref-fp64): 0.00951 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2089193Z E0601 05:10:17.208000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00191, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2094038Z E0601 05:10:17.209000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00290, (ref-fp64): 0.00429 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2100383Z E0601 05:10:17.209000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00361, (ref-fp64): 0.00493 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2105356Z E0601 05:10:17.210000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00442, (ref-fp64): 0.00557 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2110111Z E0601 05:10:17.210000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00397, (ref-fp64): 0.00563 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2114989Z E0601 05:10:17.211000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00421, (ref-fp64): 0.00601 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2119343Z E0601 05:10:17.211000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00364, (ref-fp64): 0.00662 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2132241Z E0601 05:10:17.212000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00406, (ref-fp64): 0.00697 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2137088Z E0601 05:10:17.213000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00108, (ref-fp64): 0.00274 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2149381Z E0601 05:10:17.214000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00272, (ref-fp64): 0.00457 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2154466Z E0601 05:10:17.215000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00924, (ref-fp64): 0.00949 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2164820Z E0601 05:10:17.216000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00886, (ref-fp64): 0.00915 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2169480Z E0601 05:10:17.216000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00179, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2174579Z E0601 05:10:17.217000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00287, (ref-fp64): 0.00438 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2179245Z E0601 05:10:17.217000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00352, (ref-fp64): 0.00423 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2183815Z E0601 05:10:17.217000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00448, (ref-fp64): 0.00556 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2188557Z E0601 05:10:17.218000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00383, (ref-fp64): 0.00722 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2193181Z E0601 05:10:17.218000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00476, (ref-fp64): 0.00726 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2197961Z E0601 05:10:17.219000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00410, (ref-fp64): 0.00738 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2210759Z E0601 05:10:17.220000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00446, (ref-fp64): 0.00772 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2215393Z E0601 05:10:17.221000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00149, (ref-fp64): 0.00331 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2227972Z E0601 05:10:17.222000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00287, (ref-fp64): 0.00458 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2234910Z E0601 05:10:17.223000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00226, (ref-fp64): 0.00294 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:17.2240332Z E0601 05:10:17.223000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00145 and shape=torch.Size([1024, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2406655Z E0601 05:10:17.240000 139635068625536 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00079, (ref-fp64): 0.00158 and shape=torch.Size([50304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:17.2425662Z pass 2024-06-01T05:10:17.3069083Z TIMING: entire_frame_compile:85.30161 code_gen:20.67257 inductor_compile:46.85588 backend_compile:74.20797 2024-06-01T05:10:17.3070967Z STATS: call_* op count: 883 | FakeTensorMode.__torch_dispatch__:91370 | FakeTensor.__torch_dispatch__:12825 | attempt fast:2577 | fast is_contiguous:2577 | ProxyTorchDispatchMode.__torch_dispatch__:19295 2024-06-01T05:10:17.3072375Z Dynamo produced 3 graphs covering 883 ops with 7 graph breaks (5 unique) 2024-06-01T05:10:24.9697495Z 2024-06-01T05:10:27.3437009Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:10:27.3437766Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:10:27.3438220Z cuda train nvidia_deeprecommender 2024-06-01T05:10:36.2834926Z W0601 05:10:36.282000 140379317703296 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:10:41.4244786Z E0601 05:10:41.423000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8.70142, (ref-fp64): 9.81963 and shape=torch.Size([4, 197951]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.4249961Z E0601 05:10:41.424000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03254, (ref-fp64): 0.84514 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.4254156Z E0601 05:10:41.424000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00007, (ref-fp64): 0.00005 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.4263220Z E0601 05:10:41.425000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.05003, (ref-fp64): 0.03623 and shape=torch.Size([512, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.4267624Z E0601 05:10:41.426000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04563, (ref-fp64): 0.07534 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.4700407Z E0601 05:10:41.469000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00038, (ref-fp64): 0.00043 and shape=torch.Size([197951, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.4704247Z E0601 05:10:41.469000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00014, (ref-fp64): 0.00015 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.5135037Z E0601 05:10:41.513000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00011, (ref-fp64): 0.00014 and shape=torch.Size([512, 197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5139238Z E0601 05:10:41.513000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.08289, (ref-fp64): 0.08926 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5143747Z E0601 05:10:41.513000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03445, (ref-fp64): 0.03230 and shape=torch.Size([1024, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5148493Z E0601 05:10:41.514000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00244, (ref-fp64): 0.00262 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.5153771Z E0601 05:10:41.514000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00082, (ref-fp64): 0.00064 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.5157787Z E0601 05:10:41.515000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00015, (ref-fp64): 0.00017 and shape=torch.Size([197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5162620Z E0601 05:10:41.515000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00282, (ref-fp64): 0.00334 and shape=torch.Size([512, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5167595Z E0601 05:10:41.516000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00302, (ref-fp64): 0.00339 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5599513Z E0601 05:10:41.559000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00139, (ref-fp64): 0.00155 and shape=torch.Size([197951, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.5602501Z E0601 05:10:41.559000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00198, (ref-fp64): 0.00253 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.5605903Z E0601 05:10:41.560000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00202, (ref-fp64): 0.00242 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:10:41.5610318Z E0601 05:10:41.560000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00180, (ref-fp64): 0.00185 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.6040373Z E0601 05:10:41.603000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00239, (ref-fp64): 0.00299 and shape=torch.Size([512, 197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.6044414Z E0601 05:10:41.604000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00309, (ref-fp64): 0.00339 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.6049155Z E0601 05:10:41.604000 140379317703296 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00323, (ref-fp64): 0.00326 and shape=torch.Size([1024, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:10:41.6053112Z pass 2024-06-01T05:10:41.6331744Z TIMING: entire_frame_compile:11.50244 code_gen:4.54857 inductor_compile:6.47943 backend_compile:10.36627 2024-06-01T05:10:41.6334764Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:7629 | attempt fast:212 | fast is_contiguous:212 | FakeTensor.__torch_dispatch__:1123 | ProxyTorchDispatchMode.__torch_dispatch__:1490 2024-06-01T05:10:41.6337448Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-06-01T05:10:45.4198654Z 2024-06-01T05:10:46.3528340Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:10:46.3528823Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:10:46.3529265Z cuda train opacus_cifar10 2024-06-01T05:10:46.3552596Z Traceback (most recent call last): 2024-06-01T05:10:46.3553488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model 2024-06-01T05:10:46.3554281Z self.model_iter_fn(model, example_inputs) 2024-06-01T05:10:46.3555164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 441, in forward_and_backward_pass 2024-06-01T05:10:46.3556300Z self.grad_scaler.scale(loss).backward() 2024-06-01T05:10:46.3557577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 520, in backward 2024-06-01T05:10:46.3558625Z torch.autograd.backward( 2024-06-01T05:10:46.3559748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 284, in backward 2024-06-01T05:10:46.3560827Z _engine_run_backward( 2024-06-01T05:10:46.3561804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward 2024-06-01T05:10:46.3563170Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-01T05:10:46.3564648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 72, in __call__ 2024-06-01T05:10:46.3565661Z return self.hook(module, *args, **kwargs) 2024-06-01T05:10:46.3566775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-06-01T05:10:46.3568130Z activations, backprops = self.rearrange_grad_samples( 2024-06-01T05:10:46.3569310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-06-01T05:10:46.3570262Z raise ValueError( 2024-06-01T05:10:46.3571102Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-06-01T05:10:46.3571823Z 2024-06-01T05:10:46.3572129Z The above exception was the direct cause of the following exception: 2024-06-01T05:10:46.3572596Z 2024-06-01T05:10:46.3572745Z Traceback (most recent call last): 2024-06-01T05:10:46.3575788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T05:10:46.3576658Z ) = runner.load_model( 2024-06-01T05:10:46.3577339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model 2024-06-01T05:10:46.3578118Z self.validate_model(model, example_inputs) 2024-06-01T05:10:46.3578916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model 2024-06-01T05:10:46.3579719Z raise RuntimeError("Eager run failed") from e 2024-06-01T05:10:46.3580206Z RuntimeError: Eager run failed 2024-06-01T05:10:46.3580468Z 2024-06-01T05:10:46.3580588Z eager_fail_to_run 2024-06-01T05:10:49.6244789Z 2024-06-01T05:10:50.5134188Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:10:50.5134660Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:10:50.5135120Z cuda train phlippe_densenet 2024-06-01T05:11:42.3513347Z pass 2024-06-01T05:11:42.3517859Z TIMING: entire_frame_compile:48.01988 code_gen:11.34186 inductor_compile:25.96764 backend_compile:42.44896 2024-06-01T05:11:42.3519954Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:6771 | FakeTensorMode.__torch_dispatch__:50398 | attempt fast:1467 | fast is_contiguous:1419 | ProxyTorchDispatchMode.__torch_dispatch__:11455 | slow no contiguity match:48 2024-06-01T05:11:42.3521516Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-06-01T05:11:48.1205884Z 2024-06-01T05:11:48.7911160Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:11:48.7911648Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:11:48.7912106Z cuda train phlippe_resnet 2024-06-01T05:12:08.5071972Z E0601 05:12:08.506000 140094560309888 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00275, (ref-fp64): 0.00101 and shape=torch.Size([4, 10]). res.dtype: torch.float16, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:12:08.5077362Z E0601 05:12:08.507000 140094560309888 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00102, (ref-fp64): 0.00001 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:12:08.5091488Z fail_accuracy 2024-06-01T05:12:08.5092223Z TIMING: entire_frame_compile:17.41262 code_gen:3.46249 inductor_compile:8.7485 backend_compile:15.41048 2024-06-01T05:12:08.5093749Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:2555 | FakeTensorMode.__torch_dispatch__:18761 | attempt fast:586 | fast is_contiguous:586 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-06-01T05:12:08.5095303Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-06-01T05:12:12.7331127Z 2024-06-01T05:12:14.0045047Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:12:14.0045654Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:12:14.0046120Z cuda train pytorch_CycleGAN_and_pix2pix 2024-06-01T05:12:50.7020504Z E0601 05:12:50.700000 140386359763584 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01213, (ref-fp64): 0.01057 and shape=torch.Size([1, 3, 256, 256]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:12:50.7022762Z E0601 05:12:50.701000 140386359763584 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00000, (ref-fp64): 0.00022 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:12:50.7025333Z E0601 05:12:50.702000 140386359763584 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00093, (ref-fp64): 0.00079 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:12:50.7100086Z E0601 05:12:50.709000 140386359763584 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00022, (ref-fp64): 0.00019 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:12:50.7193591Z pass 2024-06-01T05:12:50.7282153Z TIMING: entire_frame_compile:8.96808 code_gen:10.71932 inductor_compile:15.08852 backend_compile:8.25887 2024-06-01T05:12:50.7291855Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:17797 | FakeTensor.__torch_dispatch__:3110 | ProxyTorchDispatchMode.__torch_dispatch__:5123 2024-06-01T05:12:50.7293534Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-06-01T05:12:54.9540995Z 2024-06-01T05:12:56.4296901Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:12:56.4297392Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:12:56.4297882Z cuda train pytorch_stargan 2024-06-01T05:13:58.1091079Z pass 2024-06-01T05:13:58.1155541Z TIMING: entire_frame_compile:27.79314 code_gen:4.6881 inductor_compile:13.48049 backend_compile:24.35662 2024-06-01T05:13:58.1159065Z STATS: call_* op count: 61 | FakeTensorMode.__torch_dispatch__:27556 | FakeTensor.__torch_dispatch__:2601 | attempt fast:864 | fast is_contiguous:864 | ProxyTorchDispatchMode.__torch_dispatch__:5427 2024-06-01T05:13:58.1160489Z Dynamo produced 2 graphs covering 61 ops with 6 graph breaks (5 unique) 2024-06-01T05:14:03.0272389Z 2024-06-01T05:14:03.9226415Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:14:03.9226907Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:14:03.9227369Z cuda train pytorch_unet 2024-06-01T05:15:05.7484309Z W0601 05:15:05.747000 139919359246976 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:15:27.4984013Z pass_due_to_skip 2024-06-01T05:15:27.5076886Z TIMING: entire_frame_compile:50.03718 code_gen:20.45199 inductor_compile:33.14389 backend_compile:44.79993 2024-06-01T05:15:27.5079966Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:6870 | FakeTensorMode.__torch_dispatch__:42567 | attempt fast:766 | fast is_contiguous:758 | ProxyTorchDispatchMode.__torch_dispatch__:9107 | slow no contiguity match:8 2024-06-01T05:15:27.5081551Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-06-01T05:15:32.9454666Z 2024-06-01T05:15:35.4834487Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:15:35.4835020Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:15:35.4835519Z cuda train resnet152 2024-06-01T05:17:34.7542445Z W0601 05:17:34.753000 139951583220352 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:19:41.5761106Z E0601 05:19:41.575000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 10328671597717497416986647755950653440.00000, (ref-fp64): 10328671597717497416986647755950653440.00000 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5766006Z E0601 05:19:41.576000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 9743941719078621999548186634041163776.00000, (ref-fp64): 9743941719078621999548186634041163776.00000 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5772294Z E0601 05:19:41.576000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 199979232645733563279643013338365952.00000, (ref-fp64): 199979232645733563279643013338365952.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5776421Z E0601 05:19:41.577000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 568417274847048224411161864446148608.00000, (ref-fp64): 568417274847048224411161864446148608.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5780982Z E0601 05:19:41.577000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5415966509405956756603454921637888.00000, (ref-fp64): 5415966509405956756603454921637888.00000 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5794432Z E0601 05:19:41.578000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1044139720766333760023717528731648.00000, (ref-fp64): 1044139720766333760023717528731648.00000 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5798672Z E0601 05:19:41.579000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 166431222050800709645744284175958016.00000, (ref-fp64): 166431222050800709645744284175958016.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5803575Z E0601 05:19:41.579000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 230807783489513280289212651999854592.00000, (ref-fp64): 230807783489513280289212651999854592.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5807380Z E0601 05:19:41.580000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 166416098675865820164253784096112640.00000, (ref-fp64): 166416098675865820164253784096112640.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5811523Z E0601 05:19:41.580000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 287183673113477707853973626745257984.00000, (ref-fp64): 287183673113477707853973626745257984.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5816507Z E0601 05:19:41.581000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 91570321305939002474604421282004992.00000, (ref-fp64): 91570321305939002474604421282004992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5820713Z E0601 05:19:41.581000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 211948255280286472513428424180629504.00000, (ref-fp64): 211948255280286472513428424180629504.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5825048Z E0601 05:19:41.582000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 71804966642133336651798393296781312.00000, (ref-fp64): 71804966642133336651798393296781312.00000 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5829803Z E0601 05:19:41.582000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 30008632037502504733309491384156160.00000, (ref-fp64): 30008632037502504733309491384156160.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5834567Z E0601 05:19:41.583000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 71702198327976919772934597104369664.00000, (ref-fp64): 71702198327976919772934597104369664.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5838781Z E0601 05:19:41.583000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 88867857359890790931777595444822016.00000, (ref-fp64): 88867857359890790931777595444822016.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5842993Z E0601 05:19:41.583000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 91570321305939002474604421282004992.00000, (ref-fp64): 91570321305939002474604421282004992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5847537Z E0601 05:19:41.584000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 239512392471229536935003303257833472.00000, (ref-fp64): 239512392471229536935003303257833472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5851602Z E0601 05:19:41.584000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 118738845949867884790043810432286720.00000, (ref-fp64): 118738845949867884790043810432286720.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5855844Z E0601 05:19:41.585000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 212746654828475252086584269634273280.00000, (ref-fp64): 212746654828475252086584269634273280.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5859957Z E0601 05:19:41.585000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 98175779997112958651462161834967040.00000, (ref-fp64): 98175779997112958651462161834967040.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5864213Z E0601 05:19:41.586000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 149508223167153362548116408227594240.00000, (ref-fp64): 149508223167153362548116408227594240.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5868479Z E0601 05:19:41.586000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 108119457002681049990296650649698304.00000, (ref-fp64): 108119457002681049990296650649698304.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5873051Z E0601 05:19:41.586000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 171072842714012028675485437339369472.00000, (ref-fp64): 171072842714012028675485437339369472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5877928Z E0601 05:19:41.587000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 66287207322882670748578032117612544.00000, (ref-fp64): 66287207322882670748578032117612544.00000 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5882613Z E0601 05:19:41.587000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 42650582738633273063939053358940160.00000, (ref-fp64): 42650582738633273063939053358940160.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5887726Z E0601 05:19:41.588000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 41791801179119990936874225529520128.00000, (ref-fp64): 41791801179119990936874225529520128.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5892237Z E0601 05:19:41.588000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 243113709873953707490076874550280192.00000, (ref-fp64): 243113709873953707490076874550280192.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5896951Z E0601 05:19:41.589000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 648159237022147541393100816187916288.00000, (ref-fp64): 648159237022147541393100816187916288.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5901231Z E0601 05:19:41.589000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 232429160799025194942555452403089408.00000, (ref-fp64): 232429160799025194942555452403089408.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5905946Z E0601 05:19:41.590000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1142844146347974609636362202615119872.00000, (ref-fp64): 1142844146347974609636362202615119872.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5910515Z E0601 05:19:41.590000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 99826884577307123218498021977227264.00000, (ref-fp64): 99826884577307123218498021977227264.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5915193Z E0601 05:19:41.591000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 576869269788768410320904062526029824.00000, (ref-fp64): 576869269788768410320904062526029824.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5919824Z E0601 05:19:41.591000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 138181855948457015485157136972906496.00000, (ref-fp64): 138181855948457015485157136972906496.00000 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5925034Z E0601 05:19:41.592000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 129419551354975033171123711133089792.00000, (ref-fp64): 129419551354975033171123711133089792.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5929544Z E0601 05:19:41.592000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 195220794797470077854971502039924736.00000, (ref-fp64): 195220794797470077854971502039924736.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5934164Z E0601 05:19:41.592000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 143434853314860858556208890972733440.00000, (ref-fp64): 143434853314860858556208890972733440.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5938641Z E0601 05:19:41.593000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 593056065642460648488324894931550208.00000, (ref-fp64): 593056065642460648488324894931550208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5943230Z E0601 05:19:41.593000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 63744749370389567468955485327065088.00000, (ref-fp64): 63744749370389567468955485327065088.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5947666Z E0601 05:19:41.594000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 410311544695875342450158243634216960.00000, (ref-fp64): 410311544695875342450158243634216960.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5952688Z E0601 05:19:41.594000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 26849360398294471850923321100075008.00000, (ref-fp64): 26849360398294471850923321100075008.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5957338Z E0601 05:19:41.595000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 354401958036146304595788754447237120.00000, (ref-fp64): 354401958036146304595788754447237120.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5962231Z E0601 05:19:41.595000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 108629961766402135464779066621558784.00000, (ref-fp64): 108629961766402135464779066621558784.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5967591Z E0601 05:19:41.596000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 33984577910942279256350526856495104.00000, (ref-fp64): 33984577910942279256350526856495104.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.5972798Z E0601 05:19:41.596000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58372419930671224757437683644497920.00000, (ref-fp64): 58372419930671224757437683644497920.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5980013Z E0601 05:19:41.597000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 20089776908931327738101322559258624.00000, (ref-fp64): 20089776908931327738101322559258624.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5984589Z E0601 05:19:41.598000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 26849360398294471850923321100075008.00000, (ref-fp64): 26849360398294471850923321100075008.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5989150Z E0601 05:19:41.598000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 128113838418425307664980548392058880.00000, (ref-fp64): 128113838418425307664980548392058880.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5993936Z E0601 05:19:41.598000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 72750883374261358280913895118864384.00000, (ref-fp64): 72750883374261358280913895118864384.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.5998684Z E0601 05:19:41.599000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 693225112862392286316127804641509376.00000, (ref-fp64): 693225112862392286316127804641509376.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6003380Z E0601 05:19:41.599000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 43640014055397372581501352724660224.00000, (ref-fp64): 43640014055397372581501352724660224.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6008149Z E0601 05:19:41.600000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 642469918274282082845181959519338496.00000, (ref-fp64): 642469918274282082845181959519338496.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6012935Z E0601 05:19:41.600000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 23674962440417063433274149641912320.00000, (ref-fp64): 23674962440417063433274149641912320.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6017678Z E0601 05:19:41.601000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 729812822615077659727179712114458624.00000, (ref-fp64): 729812822615077659727179712114458624.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6022932Z E0601 05:19:41.601000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 80361741851523830009928158319476736.00000, (ref-fp64): 80361741851523830009928158319476736.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6027783Z E0601 05:19:41.602000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 47587727273796281791837723160477696.00000, (ref-fp64): 47587727273796281791837723160477696.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6033254Z E0601 05:19:41.602000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 98603035965782541184593969973035008.00000, (ref-fp64): 98603035965782541184593969973035008.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6037733Z E0601 05:19:41.603000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44176255347730535768407754317758464.00000, (ref-fp64): 44176255347730535768407754317758464.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6042294Z E0601 05:19:41.603000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 520548354279337512600093942463070208.00000, (ref-fp64): 520548354279337512600093942463070208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6047316Z E0601 05:19:41.604000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35734048059292696401440138646781952.00000, (ref-fp64): 35734048059292696401440138646781952.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6051846Z E0601 05:19:41.604000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 562953406214050219725514334783143936.00000, (ref-fp64): 562953406214050219725514334783143936.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6056625Z E0601 05:19:41.605000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 20762589189306056737097834026762240.00000, (ref-fp64): 20762589189306056737097834026762240.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6061171Z E0601 05:19:41.605000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 303543821438829370631330767277391872.00000, (ref-fp64): 303543821438829370631330767277391872.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6067968Z E0601 05:19:41.606000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 77467095047764073160917587482116096.00000, (ref-fp64): 77467095047764073160917587482116096.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6073093Z E0601 05:19:41.606000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 86303965935226554652225272104878080.00000, (ref-fp64): 86303965935226554652225272104878080.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6078236Z E0601 05:19:41.607000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 82483766334927989528836506451443712.00000, (ref-fp64): 82483766334927989528836506451443712.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6083141Z E0601 05:19:41.607000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 47831174232307535744755609221726208.00000, (ref-fp64): 47831174232307535744755609221726208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6087842Z E0601 05:19:41.608000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 578584102211152601279112855095869440.00000, (ref-fp64): 578584102211152601279112855095869440.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6092562Z E0601 05:19:41.608000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 18739686102097563225988255264538624.00000, (ref-fp64): 18739686102097563225988255264538624.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6097192Z E0601 05:19:41.609000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 655937233707216336958752782339276800.00000, (ref-fp64): 655937233707216336958752782339276800.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6101944Z E0601 05:19:41.609000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 14424647034969385216772934486458368.00000, (ref-fp64): 14424647034969385216772934486458368.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6106624Z E0601 05:19:41.610000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 484197296090114737211419094524362752.00000, (ref-fp64): 484197296090114737211419094524362752.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6112166Z E0601 05:19:41.610000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 88219180088227379886362758442123264.00000, (ref-fp64): 88219180088227379886362758442123264.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6117401Z E0601 05:19:41.611000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39235333780290468261399666999951360.00000, (ref-fp64): 39235333780290468261399666999951360.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6122734Z E0601 05:19:41.611000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 88067258513560509174838450634883072.00000, (ref-fp64): 88067258513560509174838450634883072.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6127585Z E0601 05:19:41.612000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35780900377027845701617023918800896.00000, (ref-fp64): 35780900377027845701617023918800896.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6132204Z E0601 05:19:41.612000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 853203111397990029397691659622285312.00000, (ref-fp64): 853203111397990029397691659622285312.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6136898Z E0601 05:19:41.613000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 18305958910815595820482074266042368.00000, (ref-fp64): 18305958910815595820482074266042368.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6141562Z E0601 05:19:41.613000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 927187523049219741281994472326955008.00000, (ref-fp64): 927187523049219741281994472326955008.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6146211Z E0601 05:19:41.614000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7283191157045600716183116389023744.00000, (ref-fp64): 7283191157045600716183116389023744.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6151124Z E0601 05:19:41.614000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 367246206806026300990712496099688448.00000, (ref-fp64): 367246206806026300990712496099688448.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6156471Z E0601 05:19:41.615000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 116508796414857193940130948202889216.00000, (ref-fp64): 116508796414857193940130948202889216.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6161343Z E0601 05:19:41.615000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 57850197204253592612365040646881280.00000, (ref-fp64): 57850197204253592612365040646881280.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6166759Z E0601 05:19:41.616000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 86817738356569021038067826286395392.00000, (ref-fp64): 86817738356569021038067826286395392.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6171464Z E0601 05:19:41.616000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 15494925045445043791909410733817856.00000, (ref-fp64): 15494925045445043791909410733817856.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6176177Z E0601 05:19:41.617000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 838779910811816962625017749365063680.00000, (ref-fp64): 838779910811816962625017749365063680.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6180807Z E0601 05:19:41.617000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7444475783777533327165203377291264.00000, (ref-fp64): 7444475783777533327165203377291264.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6185362Z E0601 05:19:41.618000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1248812984036432274371272665886359552.00000, (ref-fp64): 1248812984036432274371272665886359552.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6190181Z E0601 05:19:41.618000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4435114980132863667434728909176832.00000, (ref-fp64): 4435114980132863667434728909176832.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6194882Z E0601 05:19:41.619000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 559820228411211481007152808004157440.00000, (ref-fp64): 559820228411211481007152808004157440.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6200237Z E0601 05:19:41.619000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 64776954020411122734398565353783296.00000, (ref-fp64): 64776954020411122734398565353783296.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6205110Z E0601 05:19:41.620000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 59136872147467286533966466897674240.00000, (ref-fp64): 59136872147467286533966466897674240.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6210750Z E0601 05:19:41.620000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 102243143291033031231876293556436992.00000, (ref-fp64): 102243143291033031231876293556436992.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6215237Z E0601 05:19:41.621000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 12255576975083271680881142622322688.00000, (ref-fp64): 12255576975083271680881142622322688.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6220011Z E0601 05:19:41.621000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1009944797429725682882241505131495424.00000, (ref-fp64): 1009944797429725682882241505131495424.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6224539Z E0601 05:19:41.622000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2524273996166759519009086008983552.00000, (ref-fp64): 2524273996166759519009086008983552.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6229159Z E0601 05:19:41.622000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 616339573727521221107667254527393792.00000, (ref-fp64): 616339573727521221107667254527393792.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6234138Z E0601 05:19:41.622000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1060974210359713881391120240345088.00000, (ref-fp64): 1060974210359713881391120240345088.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6238649Z E0601 05:19:41.623000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 912756423786836159448964267419631616.00000, (ref-fp64): 912756423786836159448964267419631616.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6244014Z E0601 05:19:41.623000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 117838259954770006028208083627409408.00000, (ref-fp64): 117838259954770006028208083627409408.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6249123Z E0601 05:19:41.624000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 34943638735794087169027907266805760.00000, (ref-fp64): 34943638735794087169027907266805760.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6254297Z E0601 05:19:41.624000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 86624693115682291215518221717209088.00000, (ref-fp64): 86624693115682291215518221717209088.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6258799Z E0601 05:19:41.625000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2169297399545674799976085970747392.00000, (ref-fp64): 2169297399545674799976085970747392.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6263575Z E0601 05:19:41.625000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 633988630609249091901742153541353472.00000, (ref-fp64): 633988630609249091901742153541353472.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6268263Z E0601 05:19:41.626000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1268440301637692580670415183020032.00000, (ref-fp64): 1268440301637692580670415183020032.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6273278Z E0601 05:19:41.626000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1477821457928288239496773205732884480.00000, (ref-fp64): 1477821457928288239496773205732884480.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6278035Z E0601 05:19:41.627000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 615350534034544314219815768162304.00000, (ref-fp64): 615350534034544314219815768162304.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6282820Z E0601 05:19:41.627000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 771594982641163879415112776462368768.00000, (ref-fp64): 771594982641163879415112776462368768.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6288313Z E0601 05:19:41.628000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 119403275880472927262329124866228224.00000, (ref-fp64): 119403275880472927262329124866228224.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6293954Z E0601 05:19:41.628000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 97562760818130022182605709457227776.00000, (ref-fp64): 97562760818130022182605709457227776.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6300146Z E0601 05:19:41.629000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 174672859975132121374601362903400448.00000, (ref-fp64): 174672859975132121374601362903400448.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6304692Z E0601 05:19:41.630000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 706461476065345900879475986399232.00000, (ref-fp64): 706461476065345900879475986399232.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6309373Z E0601 05:19:41.630000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 517532112204890828039480189055401984.00000, (ref-fp64): 517532112204890828039480189055401984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6313903Z E0601 05:19:41.630000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 222461292486983289233846661283840.00000, (ref-fp64): 222461292486983289233846661283840.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6318679Z E0601 05:19:41.631000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 426301677534322155307646132349304832.00000, (ref-fp64): 426301677534322155307646132349304832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6323269Z E0601 05:19:41.631000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 68635978392252581495470985052160.00000, (ref-fp64): 68635978392252581495470985052160.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6328297Z E0601 05:19:41.632000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 265253250555013803016667409929994240.00000, (ref-fp64): 265253250555013803016667409929994240.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6333114Z E0601 05:19:41.632000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 65945373817317208445339207064354816.00000, (ref-fp64): 65945373817317208445339207064354816.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6338421Z E0601 05:19:41.633000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 13581638459375173440334339395026944.00000, (ref-fp64): 13581638459375173440334339395026944.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6343466Z E0601 05:19:41.633000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 27727670125527924333802392844763136.00000, (ref-fp64): 27727670125527924333802392844763136.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6348640Z E0601 05:19:41.634000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8927210797347807355774630715457536.00000, (ref-fp64): 8927210797347807355774630715457536.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6353245Z E0601 05:19:41.634000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 68635978392252581495470985052160.00000, (ref-fp64): 68635978392252581495470985052160.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6358191Z E0601 05:19:41.635000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58010647706461533743236822358556672.00000, (ref-fp64): 58010647706461533743236822358556672.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6362845Z E0601 05:19:41.635000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 304848586083316348671866748010496.00000, (ref-fp64): 304848586083316348671866748010496.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6368073Z E0601 05:19:41.636000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1206144284163441430539676184676925440.00000, (ref-fp64): 1206144284163441430539676184676925440.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6372308Z E0601 05:19:41.636000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 95443683214690310004069202132992.00000, (ref-fp64): 95443683214690310004069202132992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6377036Z E0601 05:19:41.637000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 925052784207817701711969171521142784.00000, (ref-fp64): 925052784207817701711969171521142784.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6381658Z E0601 05:19:41.637000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 27109495725596663035477419360256.00000, (ref-fp64): 27109495725596663035477419360256.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6386392Z E0601 05:19:41.638000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 381220461837554643308257708200689664.00000, (ref-fp64): 381220461837554643308257708200689664.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6391529Z E0601 05:19:41.638000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 71976300051686206344784460191367168.00000, (ref-fp64): 71976300051686206344784460191367168.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6396963Z E0601 05:19:41.639000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 21591780189101318254947226091520000.00000, (ref-fp64): 21591780189101318254947226091520000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6402124Z E0601 05:19:41.639000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 46333750662755690005236733307781120.00000, (ref-fp64): 46333750662755690005236733307781120.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6406759Z E0601 05:19:41.640000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2474599736247100274376704.00000, (ref-fp64): 2474599736247100274376704.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6411477Z E0601 05:19:41.640000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1225269526282335737781306320459137024.00000, (ref-fp64): 1225269526282335737781306320459137024.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6416032Z E0601 05:19:41.641000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 655413748481471274811392.00000, (ref-fp64): 655413748481471274811392.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6420756Z E0601 05:19:41.641000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 828317812917056754486388525470056448.00000, (ref-fp64): 828317812917056754486388525470056448.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6425375Z E0601 05:19:41.642000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 156864227812618264379392.00000, (ref-fp64): 156864227812618264379392.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6430303Z E0601 05:19:41.642000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 438037948125435551997376409722945536.00000, (ref-fp64): 438037948125435551997376409722945536.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6435740Z E0601 05:19:41.643000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 92040047188700659588450482210734080.00000, (ref-fp64): 92040047188700659588450482210734080.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6440810Z E0601 05:19:41.643000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44361240799154524193240741413126144.00000, (ref-fp64): 44361240799154524193240741413126144.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6446048Z E0601 05:19:41.644000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 50579269376706939901566077231955968.00000, (ref-fp64): 50579269376706939901566077231955968.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6450423Z E0601 05:19:41.644000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 566198250110690116239360.00000, (ref-fp64): 566198250110690116239360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6455182Z E0601 05:19:41.645000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 995742051911006371484678381562232832.00000, (ref-fp64): 995742051911006371484678381562232832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6459584Z E0601 05:19:41.645000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 113604111401481771417600.00000, (ref-fp64): 113604111401481771417600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6464491Z E0601 05:19:41.646000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 733660191619024577931030104360615936.00000, (ref-fp64): 733660191619024577931030104360615936.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6469085Z E0601 05:19:41.646000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 46005128277734321029120.00000, (ref-fp64): 46005128277734321029120.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6474057Z E0601 05:19:41.646000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 704498631101713372003775477480161280.00000, (ref-fp64): 704498631101713372003775477480161280.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6479196Z E0601 05:19:41.647000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 54469216733055293823178867654787072.00000, (ref-fp64): 54469216733055293823178867654787072.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6484424Z E0601 05:19:41.648000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 27280233880705138902965888787415040.00000, (ref-fp64): 27280233880705138902965888787415040.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6489631Z E0601 05:19:41.648000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 55394899309495361132557864599027712.00000, (ref-fp64): 55394899309495361132557864599027712.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6494220Z E0601 05:19:41.649000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 187879122028730817445888.00000, (ref-fp64): 187879122028730817445888.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6499121Z E0601 05:19:41.649000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1381734679248587897684957777427955712.00000, (ref-fp64): 1381734679248587897684957777427955712.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6503434Z E0601 05:19:41.649000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 32180217928385935966208.00000, (ref-fp64): 32180217928385935966208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6508253Z E0601 05:19:41.650000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 963435701443446596478618918256115712.00000, (ref-fp64): 963435701443446596478618918256115712.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6513009Z E0601 05:19:41.650000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5519711547670487105536.00000, (ref-fp64): 5519711547670487105536.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6517813Z E0601 05:19:41.651000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 431732016640994694131472074822123520.00000, (ref-fp64): 431732016640994694131472074822123520.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6522891Z E0601 05:19:41.651000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 88921601179249406953256010582589440.00000, (ref-fp64): 88921601179249406953256010582589440.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6528450Z E0601 05:19:41.652000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35607071642181887121048545504788480.00000, (ref-fp64): 35607071642181887121048545504788480.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6533340Z E0601 05:19:41.652000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44496614350694564403084532463435776.00000, (ref-fp64): 44496614350694564403084532463435776.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6537653Z E0601 05:19:41.653000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 18629489347757319651328.00000, (ref-fp64): 18629489347757319651328.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6542329Z E0601 05:19:41.653000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 958630534191807157784355126659514368.00000, (ref-fp64): 958630534191807157784355126659514368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6546854Z E0601 05:19:41.654000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2770648093575851540480.00000, (ref-fp64): 2770648093575851540480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6551524Z E0601 05:19:41.654000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 706391102370083194830486013579624448.00000, (ref-fp64): 706391102370083194830486013579624448.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6556130Z E0601 05:19:41.655000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 625886425363475202048.00000, (ref-fp64): 625886425363475202048.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6561255Z E0601 05:19:41.655000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 681470407363910921569299468948340736.00000, (ref-fp64): 681470407363910921569299468948340736.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6566427Z E0601 05:19:41.656000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 53084364619663181785882987935563776.00000, (ref-fp64): 53084364619663181785882987935563776.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6571550Z E0601 05:19:41.656000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22957133865035929587145674588160000.00000, (ref-fp64): 22957133865035929587145674588160000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6576376Z E0601 05:19:41.657000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 42873287532922945299439469304217600.00000, (ref-fp64): 42873287532922945299439469304217600.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6580788Z E0601 05:19:41.657000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1501933607326012145664.00000, (ref-fp64): 1501933607326012145664.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6585651Z E0601 05:19:41.658000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 987923881466771323795161423703703552.00000, (ref-fp64): 987923881466771323795161423703703552.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6590329Z E0601 05:19:41.658000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 197702848831546392576.00000, (ref-fp64): 197702848831546392576.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6595134Z E0601 05:19:41.659000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 917105577750224001164701132598018048.00000, (ref-fp64): 917105577750224001164701132598018048.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6599865Z E0601 05:19:41.659000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 40661187270111830016.00000, (ref-fp64): 40661187270111830016.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6604677Z E0601 05:19:41.660000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 448581091257288323512678499385081856.00000, (ref-fp64): 448581091257288323512678499385081856.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6609695Z E0601 05:19:41.660000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 49292150193612471862157293163380736.00000, (ref-fp64): 49292150193612471862157293163380736.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6615129Z E0601 05:19:41.661000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 23190320842352358250931895601725440.00000, (ref-fp64): 23190320842352358250931895601725440.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6620184Z E0601 05:19:41.661000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44902841395700913060755062051045376.00000, (ref-fp64): 44902841395700913060755062051045376.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6624758Z E0601 05:19:41.662000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 94941130967120707584.00000, (ref-fp64): 94941130967120707584.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6629750Z E0601 05:19:41.662000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 778520048942101053340179781921013760.00000, (ref-fp64): 778520048942101053340179781921013760.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6634660Z E0601 05:19:41.663000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 15065452938550108160.00000, (ref-fp64): 15065452938550108160.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6639383Z E0601 05:19:41.663000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 993034499905156422407234477549944832.00000, (ref-fp64): 993034499905156422407234477549944832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6643949Z E0601 05:19:41.663000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2991227293390027264.00000, (ref-fp64): 2991227293390027264.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6648929Z E0601 05:19:41.664000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 446805174238396809603316147843760128.00000, (ref-fp64): 446805174238396809603316147843760128.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6653804Z E0601 05:19:41.664000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35820204745206290519354857790373888.00000, (ref-fp64): 35820204745206290519354857790373888.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6659165Z E0601 05:19:41.665000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 26242222371413852582163063278403584.00000, (ref-fp64): 26242222371413852582163063278403584.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6664111Z E0601 05:19:41.665000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 56423536738571717905616793715605504.00000, (ref-fp64): 56423536738571717905616793715605504.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6668564Z E0601 05:19:41.666000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 9761206042804131840.00000, (ref-fp64): 9761206042804131840.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6673521Z E0601 05:19:41.666000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 975638141319456272406927194837745664.00000, (ref-fp64): 975638141319456272406927194837745664.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6678096Z E0601 05:19:41.667000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2083944040776746496.00000, (ref-fp64): 2083944040776746496.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6682925Z E0601 05:19:41.667000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1133471086851436008296782110119690240.00000, (ref-fp64): 1133471086851436008296782110119690240.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6687773Z E0601 05:19:41.668000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 515984119528470400.00000, (ref-fp64): 515984119528470400.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6692598Z E0601 05:19:41.668000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 649208901108355536401551728274571264.00000, (ref-fp64): 649208901108355536401551728274571264.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6697562Z E0601 05:19:41.669000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 56106304710352628114224589683294208.00000, (ref-fp64): 56106304710352628114224589683294208.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6702767Z E0601 05:19:41.669000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 36452953829377307577620644266770432.00000, (ref-fp64): 36452953829377307577620644266770432.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6707714Z E0601 05:19:41.670000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 73019984797269225180789433685770240.00000, (ref-fp64): 73019984797269225180789433685770240.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6712219Z E0601 05:19:41.670000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1838398151119407360.00000, (ref-fp64): 1838398151119407360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6717152Z E0601 05:19:41.671000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1065492841328928476940558892370231296.00000, (ref-fp64): 1065492841328928476940558892370231296.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6721477Z E0601 05:19:41.671000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 414687036166881984.00000, (ref-fp64): 414687036166881984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6726605Z E0601 05:19:41.672000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1178019208601108807664527713176125440.00000, (ref-fp64): 1178019208601108807664527713176125440.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6731135Z E0601 05:19:41.672000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 150427894780387776.00000, (ref-fp64): 150427894780387776.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6736017Z E0601 05:19:41.673000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 833177903733437451018481684487077888.00000, (ref-fp64): 833177903733437451018481684487077888.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6740913Z E0601 05:19:41.673000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 52973457740586163834247506993086464.00000, (ref-fp64): 52973457740586163834247506993086464.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6746111Z E0601 05:19:41.674000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 47729855535099244851147699864469504.00000, (ref-fp64): 47729855535099244851147699864469504.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6751266Z E0601 05:19:41.674000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 118776410185792402377985466633814016.00000, (ref-fp64): 118776410185792402377985466633814016.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6755678Z E0601 05:19:41.675000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 705413340936593920.00000, (ref-fp64): 705413340936593920.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6760604Z E0601 05:19:41.675000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1219757937954221800532823331957637120.00000, (ref-fp64): 1219757937954221800532823331957637120.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6765026Z E0601 05:19:41.676000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 193442416441849600.00000, (ref-fp64): 193442416441849600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6770086Z E0601 05:19:41.676000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 741241646549820833534612082501091328.00000, (ref-fp64): 741241646549820833534612082501091328.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6774543Z E0601 05:19:41.677000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 48638192422010224.00000, (ref-fp64): 48638192422010224.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6779505Z E0601 05:19:41.677000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 489413419485197134227090713861947392.00000, (ref-fp64): 489413419485197134227090713861947392.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6784575Z E0601 05:19:41.678000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 119600955304016717915570365909696512.00000, (ref-fp64): 119600955304016717915570365909696512.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6789791Z E0601 05:19:41.678000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 41853413276882978127558039299948544.00000, (ref-fp64): 41853413276882978127558039299948544.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6794944Z E0601 05:19:41.679000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 55013110200598805910774958786609152.00000, (ref-fp64): 55013110200598805910774958786609152.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6799395Z E0601 05:19:41.679000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 148828611324839104.00000, (ref-fp64): 148828611324839104.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6804151Z E0601 05:19:41.679000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1026797735469344278901538450768396288.00000, (ref-fp64): 1026797735469344278901538450768396288.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6808840Z E0601 05:19:41.680000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 40524644879747912.00000, (ref-fp64): 40524644879747912.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6813590Z E0601 05:19:41.680000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 832414098244617706200207345175232512.00000, (ref-fp64): 832414098244617706200207345175232512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6818137Z E0601 05:19:41.681000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 11771326786965600.00000, (ref-fp64): 11771326786965600.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6822983Z E0601 05:19:41.681000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 803004091245259627808881791861784576.00000, (ref-fp64): 803004091245259627808881791861784576.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6828048Z E0601 05:19:41.682000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 61094237208805930101772212559151104.00000, (ref-fp64): 61094237208805930101772212559151104.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6833425Z E0601 05:19:41.682000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 38514193798506989283731591256866816.00000, (ref-fp64): 38514193798506989283731591256866816.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6838513Z E0601 05:19:41.683000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 64380402552830431977234677320646656.00000, (ref-fp64): 64380402552830431977234677320646656.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6843127Z E0601 05:19:41.683000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 183351060463415975201111264985088.00000, (ref-fp64): 183351060463415975201111264985088.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6849252Z E0601 05:19:41.684000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1651568865615648215723131605757198336.00000, (ref-fp64): 1651568865615648215723131605757198336.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6852793Z E0601 05:19:41.684000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 31306028635742872545287657750528.00000, (ref-fp64): 31306028635742872545287657750528.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6857319Z E0601 05:19:41.685000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 845760673502278505770706194879479808.00000, (ref-fp64): 845760673502278505770706194879479808.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6861990Z E0601 05:19:41.685000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7815653392119151290662271320064.00000, (ref-fp64): 7815653392119151290662271320064.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6866707Z E0601 05:19:41.686000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 503246291415637058989309492480442368.00000, (ref-fp64): 503246291415637058989309492480442368.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6872062Z E0601 05:19:41.686000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 66483284859311995930426408611872768.00000, (ref-fp64): 66483284859311995930426408611872768.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6877330Z E0601 05:19:41.687000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 30259410123201918114895148097732608.00000, (ref-fp64): 30259410123201918114895148097732608.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6882242Z E0601 05:19:41.687000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 36762207743473848285779241807642624.00000, (ref-fp64): 36762207743473848285779241807642624.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6886995Z E0601 05:19:41.688000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 38708759614219424.00000, (ref-fp64): 38708759614219424.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6891777Z E0601 05:19:41.688000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1119694928764029322890348130285387776.00000, (ref-fp64): 1119694928764029322890348130285387776.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6896216Z E0601 05:19:41.689000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 9173009672628144.00000, (ref-fp64): 9173009672628144.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6901147Z E0601 05:19:41.689000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1336142673509459089650841675083284480.00000, (ref-fp64): 1336142673509459089650841675083284480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6905530Z E0601 05:19:41.690000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2663392659093653.00000, (ref-fp64): 2663392659093653.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6910461Z E0601 05:19:41.690000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 958517668785251522010669197352239104.00000, (ref-fp64): 958517668785251522010669197352239104.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6915700Z E0601 05:19:41.691000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 81967179975210601846081117839425536.00000, (ref-fp64): 81967179975210601846081117839425536.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6920831Z E0601 05:19:41.691000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 48169369542672395147801828261888000.00000, (ref-fp64): 48169369542672395147801828261888000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6926099Z E0601 05:19:41.692000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 103426182415423135533848038601654272.00000, (ref-fp64): 103426182415423135533848038601654272.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6930556Z E0601 05:19:41.692000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 11617456245545566.00000, (ref-fp64): 11617456245545566.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6935499Z E0601 05:19:41.693000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1073004649451466146832899563612798976.00000, (ref-fp64): 1073004649451466146832899563612798976.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6940034Z E0601 05:19:41.693000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2839080274027609.00000, (ref-fp64): 2839080274027609.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6944814Z E0601 05:19:41.694000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 913972944520887533208392648982790144.00000, (ref-fp64): 913972944520887533208392648982790144.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6949189Z E0601 05:19:41.694000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 537520954548085.50000, (ref-fp64): 537520954548085.43750 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6954360Z E0601 05:19:41.694000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 447719658138651838802425476068409344.00000, (ref-fp64): 447719658138651838802425476068409344.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6959347Z E0601 05:19:41.695000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 113459886133572772625076154791886848.00000, (ref-fp64): 113459886133572772625076154791886848.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6964661Z E0601 05:19:41.696000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 48152616493888015549148856242405376.00000, (ref-fp64): 48152616493888015549148856242405376.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6969905Z E0601 05:19:41.696000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 50917674200265781616995395356852224.00000, (ref-fp64): 50917674200265781616995395356852224.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6974135Z E0601 05:19:41.696000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1815744959776403.00000, (ref-fp64): 1815744959776403.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6978938Z E0601 05:19:41.697000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 989668690078257713330478616223416320.00000, (ref-fp64): 989668690078257713330478616223416320.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6983435Z E0601 05:19:41.697000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 379030787751104.62500, (ref-fp64): 379030787751104.62500 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6988351Z E0601 05:19:41.698000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 859697196247783572340902634664755200.00000, (ref-fp64): 859697196247783572340902634664755200.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.6993041Z E0601 05:19:41.698000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 63159319918890.41406, (ref-fp64): 63159319918890.39844 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.6997902Z E0601 05:19:41.699000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 478751029349101887304998743082795008.00000, (ref-fp64): 478751029349101887304998743082795008.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7003008Z E0601 05:19:41.699000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 50070228967469299466156437384724480.00000, (ref-fp64): 50070228967469299466156437384724480.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7008403Z E0601 05:19:41.700000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 28032550318696011175059749337038848.00000, (ref-fp64): 28032550318696011175059749337038848.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7013345Z E0601 05:19:41.700000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58321365317722927916138727483637760.00000, (ref-fp64): 58321365317722927916138727483637760.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7017773Z E0601 05:19:41.701000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 193084189623248.81250, (ref-fp64): 193084189623248.84375 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7022650Z E0601 05:19:41.701000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1130484205536902784987633352196489216.00000, (ref-fp64): 1130484205536902784987633352196489216.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7027213Z E0601 05:19:41.702000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 30108996070249.98828, (ref-fp64): 30108996070249.96484 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7031996Z E0601 05:19:41.702000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 833553742644266646942947969092026368.00000, (ref-fp64): 833553742644266646942947969092026368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7036610Z E0601 05:19:41.703000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5239620925253.29785, (ref-fp64): 5239620925253.27246 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7041446Z E0601 05:19:41.703000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 419079920674163822196439831411687424.00000, (ref-fp64): 419079920674163822196439831411687424.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7046794Z E0601 05:19:41.704000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 60565356753394403593745789840523264.00000, (ref-fp64): 60565356753394403593745789840523264.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7051865Z E0601 05:19:41.704000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 32351185623669125922310435171729408.00000, (ref-fp64): 32351185623669125922310435171729408.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7056694Z E0601 05:19:41.705000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 47785593357169434395319729538465792.00000, (ref-fp64): 47785593357169434395319729538465792.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7061244Z E0601 05:19:41.705000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 16108271953228.61523, (ref-fp64): 16108271953228.61523 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7066070Z E0601 05:19:41.706000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1225041984349406491276679114834051072.00000, (ref-fp64): 1225041984349406491276679114834051072.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7070704Z E0601 05:19:41.706000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3685134774439.15967, (ref-fp64): 3685134774439.20312 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7075701Z E0601 05:19:41.707000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1336152584040860449353210238286692352.00000, (ref-fp64): 1336152584040860449353210238286692352.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7080073Z E0601 05:19:41.707000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 615967102562.07214, (ref-fp64): 615967102562.05493 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7085101Z E0601 05:19:41.708000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 528011995069101775591010145381908480.00000, (ref-fp64): 528011995069101775591010145381908480.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7090270Z E0601 05:19:41.708000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 66010773133770402510588614920896512.00000, (ref-fp64): 66010773133770402510588614920896512.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7095535Z E0601 05:19:41.709000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 55990095013871970559257082687651840.00000, (ref-fp64): 55990095013871970559257082687651840.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7100430Z E0601 05:19:41.709000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 67062119026629042630838812883812352.00000, (ref-fp64): 67062119026629042630838812883812352.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7104861Z E0601 05:19:41.710000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1546600514630.26562, (ref-fp64): 1546600514630.20605 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7109688Z E0601 05:19:41.710000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 857670844378582818532927654890307584.00000, (ref-fp64): 857670844378582818532927654890307584.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7114401Z E0601 05:19:41.711000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 257562494159.63550, (ref-fp64): 257562494159.55258 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7119091Z E0601 05:19:41.711000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 781013029975301092580172099561193472.00000, (ref-fp64): 781013029975301092580172099561193472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7123701Z E0601 05:19:41.711000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22772086977.33756, (ref-fp64): 22772086977.34031 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7128769Z E0601 05:19:41.712000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 321320504747965539223573830370852864.00000, (ref-fp64): 321320504747965539223573830370852864.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7133627Z E0601 05:19:41.712000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 43778329288644633905902831209021440.00000, (ref-fp64): 43778329288644633905902831209021440.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7139065Z E0601 05:19:41.713000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 28305912775687783660680007659814912.00000, (ref-fp64): 28305912775687783660680007659814912.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7143954Z E0601 05:19:41.713000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 31744759224723600387601605190483968.00000, (ref-fp64): 31744759224723600387601605190483968.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7148377Z E0601 05:19:41.714000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 51766933444.80675, (ref-fp64): 51766933444.79954 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7153342Z E0601 05:19:41.714000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 848328993195001007899427780312432640.00000, (ref-fp64): 848328993195001007899427780312432640.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7157970Z E0601 05:19:41.715000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 12468398298.43498, (ref-fp64): 12468398298.42069 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7162756Z E0601 05:19:41.715000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1158786466148279983908003978391060480.00000, (ref-fp64): 1158786466148279983908003978391060480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7167393Z E0601 05:19:41.716000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2817519624.24043, (ref-fp64): 2817519624.23087 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7172091Z E0601 05:19:41.716000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 678329179216136984250506922520739840.00000, (ref-fp64): 678329179216136984250506922520739840.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7177279Z E0601 05:19:41.717000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 31704828677486227262080960778207232.00000, (ref-fp64): 31704828677486227262080960778207232.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7182425Z E0601 05:19:41.717000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44091961357486099240958937238863872.00000, (ref-fp64): 44091961357486099240958937238863872.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7187430Z E0601 05:19:41.718000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 90841863715252409928527808953319424.00000, (ref-fp64): 90841863715252409928527808953319424.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7192592Z E0601 05:19:41.718000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8342620961.14807, (ref-fp64): 8342620961.12692 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7197531Z E0601 05:19:41.719000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1299849711782005396486631293643128832.00000, (ref-fp64): 1299849711782005396486631293643128832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7201595Z E0601 05:19:41.719000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1618477502.90264, (ref-fp64): 1618477502.86629 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7206853Z E0601 05:19:41.720000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1444055492447421527083936775864844288.00000, (ref-fp64): 1444055492447421527083936775864844288.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7211336Z E0601 05:19:41.720000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 321148771.57848, (ref-fp64): 321148771.57401 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7215937Z E0601 05:19:41.721000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 596348652855786978467835232888291328.00000, (ref-fp64): 596348652855786978467835232888291328.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7221034Z E0601 05:19:41.721000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 77768920856811338407873453163020288.00000, (ref-fp64): 77768920856811338407873453163020288.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7226132Z E0601 05:19:41.722000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 49809114685426053514897328969875456.00000, (ref-fp64): 49809114685426053514897328969875456.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7231222Z E0601 05:19:41.722000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 79602655833809340158669535943065600.00000, (ref-fp64): 79602655833809340158669535943065600.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7235721Z E0601 05:19:41.723000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 799766708.90303, (ref-fp64): 799766708.87453 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7240527Z E0601 05:19:41.723000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 846330024107358438922443937638514688.00000, (ref-fp64): 846330024107358438922443937638514688.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7245111Z E0601 05:19:41.724000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 169017831.97254, (ref-fp64): 169017831.91381 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7249997Z E0601 05:19:41.724000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1051464272845668043296454504280489984.00000, (ref-fp64): 1051464272845668043296454504280489984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7254568Z E0601 05:19:41.725000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 45840474.58813, (ref-fp64): 45840474.59743 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7259548Z E0601 05:19:41.725000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 807657559490865540995749532944826368.00000, (ref-fp64): 807657559490865540995749532944826368.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7264467Z E0601 05:19:41.726000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 49963477635658453624141620338229248.00000, (ref-fp64): 49963477635658453624141620338229248.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7269744Z E0601 05:19:41.726000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 32982588733458169944398342039535616.00000, (ref-fp64): 32982588733458169944398342039535616.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7274998Z E0601 05:19:41.727000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 82848660824886751673043069630939136.00000, (ref-fp64): 82848660824886751673043069630939136.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7279258Z E0601 05:19:41.727000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 136144015.28834, (ref-fp64): 136144015.31558 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7284179Z E0601 05:19:41.727000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 736489110936374467106475597107298304.00000, (ref-fp64): 736489110936374467106475597107298304.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7288746Z E0601 05:19:41.728000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 27246041.40691, (ref-fp64): 27246041.44413 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7294250Z E0601 05:19:41.728000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 956364289562406020425371703429824512.00000, (ref-fp64): 956364289562406020425371703429824512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7299745Z E0601 05:19:41.729000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7966555.32184, (ref-fp64): 7966555.30886 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7304769Z E0601 05:19:41.730000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 776698031728042896515405538066956288.00000, (ref-fp64): 776698031728042896515405538066956288.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7309823Z E0601 05:19:41.730000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 74750701426893030622038086727499776.00000, (ref-fp64): 74750701426893030622038086727499776.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7315200Z E0601 05:19:41.731000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39901043135052072775601839799271424.00000, (ref-fp64): 39901043135052072775601839799271424.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7320210Z E0601 05:19:41.731000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 79413297369165445656493104463609856.00000, (ref-fp64): 79413297369165445656493104463609856.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7324751Z E0601 05:19:41.732000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22436867422799894648932026810368.00000, (ref-fp64): 22436867422799894648932026810368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7329542Z E0601 05:19:41.732000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 838716291624339008926183523029614592.00000, (ref-fp64): 838716291624339008926183523029614592.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7334146Z E0601 05:19:41.732000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3839113020754793980609804894208.00000, (ref-fp64): 3839113020754793980609804894208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7338815Z E0601 05:19:41.733000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 908533536220489530360839022401552384.00000, (ref-fp64): 908533536220489530360839022401552384.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7343524Z E0601 05:19:41.733000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1106567688013340808645491818496.00000, (ref-fp64): 1106567688013340808645491818496.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7348188Z E0601 05:19:41.734000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 458354600860767809764525159194034176.00000, (ref-fp64): 458354600860767809764525159194034176.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7353205Z E0601 05:19:41.734000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 46444137670002163791608214726901760.00000, (ref-fp64): 46444137670002163791608214726901760.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7358847Z E0601 05:19:41.735000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 17434873709154349035786287385149440.00000, (ref-fp64): 17434873709154349035786287385149440.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7363753Z E0601 05:19:41.735000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 45102900421403704639333530091913216.00000, (ref-fp64): 45102900421403704639333530091913216.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7368284Z E0601 05:19:41.736000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 25129012.27010, (ref-fp64): 25129012.29314 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7373039Z E0601 05:19:41.736000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 767687114747617552143811300445650944.00000, (ref-fp64): 767687114747617552143811300445650944.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7377517Z E0601 05:19:41.737000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4865912.22528, (ref-fp64): 4865912.28442 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7382469Z E0601 05:19:41.737000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 988955017571954056124965876651786240.00000, (ref-fp64): 988955017571954056124965876651786240.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7386834Z E0601 05:19:41.738000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1643572.95452, (ref-fp64): 1643572.98514 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7392263Z E0601 05:19:41.738000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 594388958609800834331725869782925312.00000, (ref-fp64): 594388958609800834331725869782925312.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7397182Z E0601 05:19:41.739000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 55853359888312404266250002478661632.00000, (ref-fp64): 55853359888312404266250002478661632.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7402129Z E0601 05:19:41.739000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35874698391258456971956689011998720.00000, (ref-fp64): 35874698391258456971956689011998720.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7407545Z E0601 05:19:41.740000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 80167853586892354628767043733684224.00000, (ref-fp64): 80167853586892354628767043733684224.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7411784Z E0601 05:19:41.740000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5116232.88732, (ref-fp64): 5116232.90178 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7416881Z E0601 05:19:41.741000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 941668792588830439856081023031836672.00000, (ref-fp64): 941668792588830439856081023031836672.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7421258Z E0601 05:19:41.741000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1288700.39725, (ref-fp64): 1288700.46821 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7426176Z E0601 05:19:41.742000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1462815185603927402255617514205609984.00000, (ref-fp64): 1462815185603927402255617514205609984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7430755Z E0601 05:19:41.742000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 310389.44054, (ref-fp64): 310389.45322 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7435898Z E0601 05:19:41.743000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 504266866468943654178810971851587584.00000, (ref-fp64): 504266866468943654178810971851587584.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7440962Z E0601 05:19:41.743000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 73041522803288847139799671058202624.00000, (ref-fp64): 73041522803288847139799671058202624.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7446311Z E0601 05:19:41.744000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 56823428535204210502773957740986368.00000, (ref-fp64): 56823428535204210502773957740986368.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7451430Z E0601 05:19:41.744000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 87248080098702910782807707379302400.00000, (ref-fp64): 87248080098702910782807707379302400.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7455641Z E0601 05:19:41.745000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 839922.01420, (ref-fp64): 839922.15550 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7460478Z E0601 05:19:41.745000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 882730838902755365147519514690191360.00000, (ref-fp64): 882730838902755365147519514690191360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7464749Z E0601 05:19:41.746000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 158234.88580, (ref-fp64): 158234.99030 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7469772Z E0601 05:19:41.746000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 997322109187168842764469713539956736.00000, (ref-fp64): 997322109187168842764469713539956736.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7474347Z E0601 05:19:41.747000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39573.14143, (ref-fp64): 39573.19078 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7479260Z E0601 05:19:41.747000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 722353475696904049947299208201502720.00000, (ref-fp64): 722353475696904049947299208201502720.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7484558Z E0601 05:19:41.747000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 65562810630065473372905447744339968.00000, (ref-fp64): 65562810630065473372905447744339968.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7489934Z E0601 05:19:41.748000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 40277082025714552882885854876401664.00000, (ref-fp64): 40277082025714552882885854876401664.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7494873Z E0601 05:19:41.749000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 75887356551688647520114577011376128.00000, (ref-fp64): 75887356551688647520114577011376128.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7499162Z E0601 05:19:41.749000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 91822.50810, (ref-fp64): 91822.58126 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7504145Z E0601 05:19:41.749000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 856724012486189534415817997283229696.00000, (ref-fp64): 856724012486189534415817997283229696.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7508421Z E0601 05:19:41.750000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 21733.26002, (ref-fp64): 21733.18567 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7513461Z E0601 05:19:41.750000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1226969190860660605139068696666308608.00000, (ref-fp64): 1226969190860660605139068696666308608.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7517951Z E0601 05:19:41.751000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7868.80075, (ref-fp64): 7868.80875 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7522970Z E0601 05:19:41.751000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 673108421164364312818352424688287744.00000, (ref-fp64): 673108421164364312818352424688287744.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7528209Z E0601 05:19:41.752000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 66276701737543694419967538148409344.00000, (ref-fp64): 66276701737543694419967538148409344.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7533285Z E0601 05:19:41.752000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 45816052036161843796534322551324672.00000, (ref-fp64): 45816052036161843796534322551324672.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7538398Z E0601 05:19:41.753000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 136588162200073310329925485830602752.00000, (ref-fp64): 136588162200073310329925485830602752.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7542731Z E0601 05:19:41.753000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22753.86219, (ref-fp64): 22753.83724 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7547646Z E0601 05:19:41.754000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1114580152243760967146404367917121536.00000, (ref-fp64): 1114580152243760967146404367917121536.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7551929Z E0601 05:19:41.754000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4728.16600, (ref-fp64): 4728.18202 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7557138Z E0601 05:19:41.755000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 932305177095301615640120245216083968.00000, (ref-fp64): 932305177095301615640120245216083968.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7561402Z E0601 05:19:41.755000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1410.11299, (ref-fp64): 1410.11046 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7566758Z E0601 05:19:41.756000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 771624476355700286625720379113472000.00000, (ref-fp64): 771624476355700286625720379113472000.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7571709Z E0601 05:19:41.756000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 92808425232999125587556707732553728.00000, (ref-fp64): 92808425232999125587556707732553728.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7576882Z E0601 05:19:41.757000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 43799283046217300766220975819718656.00000, (ref-fp64): 43799283046217300766220975819718656.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7581860Z E0601 05:19:41.757000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 77902852386325130646031461427183616.00000, (ref-fp64): 77902852386325130646031461427183616.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7586070Z E0601 05:19:41.758000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3412.09717, (ref-fp64): 3412.19180 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7591170Z E0601 05:19:41.758000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 803964599329983653860700547312517120.00000, (ref-fp64): 803964599329983653860700547312517120.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7595566Z E0601 05:19:41.759000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 794.07781, (ref-fp64): 794.17198 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7600390Z E0601 05:19:41.759000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 994053231901587030091598787594158080.00000, (ref-fp64): 994053231901587030091598787594158080.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7604890Z E0601 05:19:41.760000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 227.31333, (ref-fp64): 227.32937 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7609911Z E0601 05:19:41.760000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 806785814455352253055265390198784000.00000, (ref-fp64): 806785814455352253055265390198784000.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7614910Z E0601 05:19:41.761000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 70837816917400536060901645053067264.00000, (ref-fp64): 70837816917400536060901645053067264.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7620174Z E0601 05:19:41.761000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 41755892251916083752096834936373248.00000, (ref-fp64): 41755892251916083752096834936373248.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7625160Z E0601 05:19:41.762000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 100271412272271545253483265166147584.00000, (ref-fp64): 100271412272271545253483265166147584.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7629890Z E0601 05:19:41.762000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2697084545334326633824834813952.00000, (ref-fp64): 2697084545334326633824834813952.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7634757Z E0601 05:19:41.763000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 806403064891016485067705442918989824.00000, (ref-fp64): 806403064891016485067705442918989824.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7639343Z E0601 05:19:41.763000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 402550930254345663095236460544.00000, (ref-fp64): 402550930254345663095236460544.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7644021Z E0601 05:19:41.763000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 770381274110288275223215600725655552.00000, (ref-fp64): 770381274110288275223215600725655552.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7648817Z E0601 05:19:41.764000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 65214877289651643402663493632.00000, (ref-fp64): 65214877289651643402663493632.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7653481Z E0601 05:19:41.764000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 403462996849529689073172577786003456.00000, (ref-fp64): 403462996849529689073172577786003456.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7658362Z E0601 05:19:41.765000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39494949436539066375952744048492544.00000, (ref-fp64): 39494949436539066375952744048492544.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7663652Z E0601 05:19:41.765000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 19939703285302644774809206240837632.00000, (ref-fp64): 19939703285302644774809206240837632.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7668587Z E0601 05:19:41.766000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 40716726779866037392613748852654080.00000, (ref-fp64): 40716726779866037392613748852654080.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7673308Z E0601 05:19:41.766000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 208063969002249827363734945792.00000, (ref-fp64): 208063969002249827363734945792.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7677998Z E0601 05:19:41.767000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 871655981521272174727117236758118400.00000, (ref-fp64): 871655981521272174727117236758118400.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7682533Z E0601 05:19:41.767000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 42682298671877203270679134208.00000, (ref-fp64): 42682298671877203270679134208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7687533Z E0601 05:19:41.768000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1045084248796786881273478946129707008.00000, (ref-fp64): 1045084248796786881273478946129707008.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7692044Z E0601 05:19:41.768000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 10225032162806111728822124544.00000, (ref-fp64): 10225032162806111728822124544.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7696807Z E0601 05:19:41.769000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 473848005397685379598959445767356416.00000, (ref-fp64): 473848005397685379598959445767356416.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7701796Z E0601 05:19:41.769000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44587153138679443128252480832405504.00000, (ref-fp64): 44587153138679443128252480832405504.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7707128Z E0601 05:19:41.770000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 36944192896597056935129887964921856.00000, (ref-fp64): 36944192896597056935129887964921856.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7712105Z E0601 05:19:41.770000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 55146777983958046126810184789524480.00000, (ref-fp64): 55146777983958046126810184789524480.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7716757Z E0601 05:19:41.771000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 27443960825029650962309971968.00000, (ref-fp64): 27443960825029650962309971968.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7721604Z E0601 05:19:41.771000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1057206361871764728073585280201785344.00000, (ref-fp64): 1057206361871764728073585280201785344.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7726242Z E0601 05:19:41.772000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4389239936061012311971725312.00000, (ref-fp64): 4389239936061012311971725312.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7730951Z E0601 05:19:41.772000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1038490380690512170917020351879708672.00000, (ref-fp64): 1038490380690512170917020351879708672.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7735529Z E0601 05:19:41.773000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 628930793937539394934145024.00000, (ref-fp64): 628930793937539394934145024.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7740182Z E0601 05:19:41.773000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 369049690499509918795301773056671744.00000, (ref-fp64): 369049690499509918795301773056671744.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7745221Z E0601 05:19:41.774000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 42831236547709945689045120401276928.00000, (ref-fp64): 42831236547709945689045120401276928.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7750530Z E0601 05:19:41.774000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 23416171278680979267326554325123072.00000, (ref-fp64): 23416171278680979267326554325123072.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7755659Z E0601 05:19:41.775000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 43353681825085437334258265429639168.00000, (ref-fp64): 43353681825085437334258265429639168.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7760117Z E0601 05:19:41.775000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2204634831438182270938120192.00000, (ref-fp64): 2204634831438182270938120192.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7765064Z E0601 05:19:41.776000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1097525900572439942348739020792528896.00000, (ref-fp64): 1097525900572439942348739020792528896.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7769700Z E0601 05:19:41.776000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 262889382673569719548641280.00000, (ref-fp64): 262889382673569719548641280.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7774379Z E0601 05:19:41.777000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 823076256447521401911075536585621504.00000, (ref-fp64): 823076256447521401911075536585621504.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7778964Z E0601 05:19:41.777000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 38172867593527516188377088.00000, (ref-fp64): 38172867593527516188377088.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7783726Z E0601 05:19:41.777000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 470387013579192995019915796693385216.00000, (ref-fp64): 470387013579192995019915796693385216.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7788821Z E0601 05:19:41.778000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39014997152690417124141913683787776.00000, (ref-fp64): 39014997152690417124141913683787776.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7794302Z E0601 05:19:41.778000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 20015656584716282028025466582990848.00000, (ref-fp64): 20015656584716282028025466582990848.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7799305Z E0601 05:19:41.779000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 39773379167336401590236369664344064.00000, (ref-fp64): 39773379167336401590236369664344064.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7803841Z E0601 05:19:41.779000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 104615717539773008291299328.00000, (ref-fp64): 104615717539773008291299328.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7808940Z E0601 05:19:41.780000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1206399304361224419820770247196016640.00000, (ref-fp64): 1206399304361224419820770247196016640.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7813276Z E0601 05:19:41.780000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 12050829915725213691019264.00000, (ref-fp64): 12050829915725213691019264.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7818065Z E0601 05:19:41.781000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1006439677330553174814140065615183872.00000, (ref-fp64): 1006439677330553174814140065615183872.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7822551Z E0601 05:19:41.781000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2247426468373113914523648.00000, (ref-fp64): 2247426468373113914523648.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7827309Z E0601 05:19:41.782000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 521985751320857161356998494943969280.00000, (ref-fp64): 521985751320857161356998494943969280.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7832529Z E0601 05:19:41.782000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 42322033481779692550167740641443840.00000, (ref-fp64): 42322033481779692550167740641443840.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7837777Z E0601 05:19:41.783000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 19362725474465412644163179046240256.00000, (ref-fp64): 19362725474465412644163179046240256.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7842774Z E0601 05:19:41.783000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 47645905660875342296717171256983552.00000, (ref-fp64): 47645905660875342296717171256983552.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7847704Z E0601 05:19:41.784000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8290071516999408722903040.00000, (ref-fp64): 8290071516999408722903040.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7852348Z E0601 05:19:41.784000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1080726716816063182573558357915860992.00000, (ref-fp64): 1080726716816063182573558357915860992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7856857Z E0601 05:19:41.785000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1720549850897383361085440.00000, (ref-fp64): 1720549850897383361085440.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7861693Z E0601 05:19:41.785000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 839552269389968101475220984644501504.00000, (ref-fp64): 839552269389968101475220984644501504.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7866053Z E0601 05:19:41.786000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 430574336249628205252608.00000, (ref-fp64): 430574336249628205252608.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7871064Z E0601 05:19:41.786000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 439625984212376172365757745155014656.00000, (ref-fp64): 439625984212376172365757745155014656.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7876126Z E0601 05:19:41.787000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 53772863595263445289631514840006656.00000, (ref-fp64): 53772863595263445289631514840006656.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7881336Z E0601 05:19:41.787000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 36270639141269013130725976711364608.00000, (ref-fp64): 36270639141269013130725976711364608.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7886424Z E0601 05:19:41.788000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 62433524227627065809618512392486912.00000, (ref-fp64): 62433524227627065809618512392486912.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7890856Z E0601 05:19:41.788000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 257.34460, (ref-fp64): 257.42006 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7895767Z E0601 05:19:41.789000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 345925374086881664211545351767719936.00000, (ref-fp64): 345925374086881664211545351767719936.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7900081Z E0601 05:19:41.789000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 38.78103, (ref-fp64): 38.76776 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7905060Z E0601 05:19:41.790000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 286027913821064859100809308487548928.00000, (ref-fp64): 286027913821064859100809308487548928.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7909504Z E0601 05:19:41.790000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.59223, (ref-fp64): 1.59400 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7914599Z E0601 05:19:41.791000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 41074089939544340085983810347335680.00000, (ref-fp64): 41074089939544340085983810347335680.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7919819Z E0601 05:19:41.791000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 46406745618435589367546100972519424.00000, (ref-fp64): 46406745618435589367546100972519424.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7932348Z E0601 05:19:41.792000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 9133974275442376847740455400505344.00000, (ref-fp64): 9133974275442376847740455400505344.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7940570Z E0601 05:19:41.793000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 11647972941413937685548835998269440.00000, (ref-fp64): 11647972941413937685548835998269440.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7952099Z E0601 05:19:41.794000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1342491099179038705368907552653312.00000, (ref-fp64): 1342491099179038705368907552653312.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7956680Z E0601 05:19:41.795000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.59223, (ref-fp64): 1.59400 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7961575Z E0601 05:19:41.795000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4796739043464957014143572021084160.00000, (ref-fp64): 4796739043464957014143572021084160.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7966099Z E0601 05:19:41.796000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 8.09240, (ref-fp64): 8.13382 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7971033Z E0601 05:19:41.796000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 419717240651638446112849528612716544.00000, (ref-fp64): 419717240651638446112849528612716544.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7975396Z E0601 05:19:41.797000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.89523, (ref-fp64): 1.93525 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7980404Z E0601 05:19:41.797000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 450090267235301037151641116619046912.00000, (ref-fp64): 450090267235301037151641116619046912.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.7984614Z E0601 05:19:41.798000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.11378, (ref-fp64): 0.12348 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7989655Z E0601 05:19:41.798000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 138582444865557941846825831652917248.00000, (ref-fp64): 138582444865557941846825831652917248.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.7996373Z E0601 05:19:41.799000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 13124673686888989399768932026417152.00000, (ref-fp64): 13124673686888989399768932026417152.00000 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8008895Z E0601 05:19:41.800000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 10633438256433389231171095278649344.00000, (ref-fp64): 10633438256433389231171095278649344.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8015247Z E0601 05:19:41.801000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 31189475496686633046139883952799744.00000, (ref-fp64): 31189475496686633046139883952799744.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8019641Z E0601 05:19:41.801000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.86629, (ref-fp64): 0.90746 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8024540Z E0601 05:19:41.802000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 975794775745105765840914670969421824.00000, (ref-fp64): 975794775745105765840914670969421824.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8028751Z E0601 05:19:41.802000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.23809, (ref-fp64): 0.24861 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8033764Z E0601 05:19:41.802000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 455035582889973519510484832848183296.00000, (ref-fp64): 455035582889973519510484832848183296.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8038150Z E0601 05:19:41.803000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00626, (ref-fp64): 0.00555 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8043064Z E0601 05:19:41.803000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 16766609365298803616376214849060864.00000, (ref-fp64): 16766609365298803616376214849060864.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8049973Z E0601 05:19:41.804000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 44339628982455887873624647011401728.00000, (ref-fp64): 44339628982455887873624647011401728.00000 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8062356Z E0601 05:19:41.805000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22597136556404823367199042280882176.00000, (ref-fp64): 22597136556404823367199042280882176.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8068781Z E0601 05:19:41.806000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 15073195581965055047219014256295936.00000, (ref-fp64): 15073195581965055047219014256295936.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8075127Z E0601 05:19:41.807000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01351, (ref-fp64): 0.01358 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8079814Z E0601 05:19:41.807000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01492, (ref-fp64): 0.01309 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8085235Z E0601 05:19:41.808000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01045, (ref-fp64): 0.01051 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8099368Z E0601 05:19:41.809000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00202, (ref-fp64): 0.00309 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8104124Z E0601 05:19:41.810000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01054, (ref-fp64): 0.01049 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8108998Z E0601 05:19:41.810000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01049, (ref-fp64): 0.01015 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8114399Z E0601 05:19:41.811000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01221, (ref-fp64): 0.01246 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8119047Z E0601 05:19:41.811000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01111, (ref-fp64): 0.01188 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8124024Z E0601 05:19:41.812000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01166, (ref-fp64): 0.01297 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8129401Z E0601 05:19:41.812000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01040, (ref-fp64): 0.01158 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8134314Z E0601 05:19:41.813000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00900, (ref-fp64): 0.00862 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8139397Z E0601 05:19:41.813000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00762, (ref-fp64): 0.00741 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8144470Z E0601 05:19:41.814000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00876, (ref-fp64): 0.00965 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8149623Z E0601 05:19:41.814000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00968, (ref-fp64): 0.01057 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8154821Z E0601 05:19:41.815000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01166, (ref-fp64): 0.01297 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8159640Z E0601 05:19:41.815000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01148, (ref-fp64): 0.01189 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8164813Z E0601 05:19:41.816000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01029, (ref-fp64): 0.01139 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8169826Z E0601 05:19:41.816000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01076, (ref-fp64): 0.01387 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8174947Z E0601 05:19:41.817000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01238, (ref-fp64): 0.01111 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8179780Z E0601 05:19:41.817000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01108, (ref-fp64): 0.01236 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8184765Z E0601 05:19:41.818000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01292, (ref-fp64): 0.01423 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8189706Z E0601 05:19:41.818000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01183, (ref-fp64): 0.01256 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8195077Z E0601 05:19:41.819000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00775, (ref-fp64): 0.00818 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8200136Z E0601 05:19:41.819000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00804, (ref-fp64): 0.00764 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8205583Z E0601 05:19:41.820000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00939, (ref-fp64): 0.01017 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8210497Z E0601 05:19:41.820000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00908, (ref-fp64): 0.01585 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8215465Z E0601 05:19:41.821000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00984, (ref-fp64): 0.01338 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8220611Z E0601 05:19:41.821000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01143, (ref-fp64): 0.01483 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8225484Z E0601 05:19:41.822000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01240, (ref-fp64): 0.01483 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8230456Z E0601 05:19:41.822000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01408, (ref-fp64): 0.01493 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8235626Z E0601 05:19:41.823000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01272, (ref-fp64): 0.01403 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8240526Z E0601 05:19:41.823000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00853, (ref-fp64): 0.01185 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8246158Z E0601 05:19:41.824000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00940, (ref-fp64): 0.01034 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8251249Z E0601 05:19:41.824000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01098, (ref-fp64): 0.01133 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8256150Z E0601 05:19:41.825000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01417, (ref-fp64): 0.01521 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8261085Z E0601 05:19:41.825000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01361, (ref-fp64): 0.01641 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8266073Z E0601 05:19:41.826000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01620, (ref-fp64): 0.01671 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8271160Z E0601 05:19:41.826000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01550, (ref-fp64): 0.01608 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8276321Z E0601 05:19:41.827000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01321, (ref-fp64): 0.01432 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8281270Z E0601 05:19:41.827000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01191, (ref-fp64): 0.01353 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8286817Z E0601 05:19:41.828000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01119, (ref-fp64): 0.01255 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8292180Z E0601 05:19:41.828000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01074, (ref-fp64): 0.01150 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8297532Z E0601 05:19:41.829000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01038, (ref-fp64): 0.01099 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8302781Z E0601 05:19:41.829000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01043, (ref-fp64): 0.01104 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8307821Z E0601 05:19:41.830000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01321, (ref-fp64): 0.01432 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8313508Z E0601 05:19:41.830000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01289, (ref-fp64): 0.01330 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8317789Z E0601 05:19:41.831000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01294, (ref-fp64): 0.01425 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8322816Z E0601 05:19:41.831000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01170, (ref-fp64): 0.01374 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8328033Z E0601 05:19:41.832000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01341, (ref-fp64): 0.01436 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8333002Z E0601 05:19:41.832000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01180, (ref-fp64): 0.01283 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8337925Z E0601 05:19:41.833000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01331, (ref-fp64): 0.01460 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8342945Z E0601 05:19:41.833000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01188, (ref-fp64): 0.01345 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8348417Z E0601 05:19:41.834000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.00965 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8354149Z E0601 05:19:41.835000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00886, (ref-fp64): 0.00970 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8359498Z E0601 05:19:41.835000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00905, (ref-fp64): 0.00974 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8364550Z E0601 05:19:41.836000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01146, (ref-fp64): 0.01346 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8369725Z E0601 05:19:41.836000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01141, (ref-fp64): 0.01288 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8374573Z E0601 05:19:41.837000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01271, (ref-fp64): 0.01523 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8379665Z E0601 05:19:41.837000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01275, (ref-fp64): 0.01496 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8384466Z E0601 05:19:41.838000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01351, (ref-fp64): 0.01431 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8389366Z E0601 05:19:41.838000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01236, (ref-fp64): 0.01387 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8396921Z E0601 05:19:41.839000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00846, (ref-fp64): 0.00974 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8401930Z E0601 05:19:41.839000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00822, (ref-fp64): 0.00948 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8407544Z E0601 05:19:41.840000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00977, (ref-fp64): 0.01053 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8412457Z E0601 05:19:41.840000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01297, (ref-fp64): 0.01582 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8417438Z E0601 05:19:41.841000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01308, (ref-fp64): 0.01580 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8422374Z E0601 05:19:41.841000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01352, (ref-fp64): 0.01574 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8427318Z E0601 05:19:41.842000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01400, (ref-fp64): 0.01630 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8432538Z E0601 05:19:41.842000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01377, (ref-fp64): 0.01450 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8437322Z E0601 05:19:41.843000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01238, (ref-fp64): 0.01340 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8442718Z E0601 05:19:41.843000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00975, (ref-fp64): 0.01156 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8448240Z E0601 05:19:41.844000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.01018 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8463293Z E0601 05:19:41.844000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00965, (ref-fp64): 0.01032 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8465533Z E0601 05:19:41.845000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01423, (ref-fp64): 0.01400 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8467625Z E0601 05:19:41.845000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01343, (ref-fp64): 0.01298 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8469715Z E0601 05:19:41.846000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01298, (ref-fp64): 0.01414 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8474045Z E0601 05:19:41.846000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01254, (ref-fp64): 0.01444 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8478655Z E0601 05:19:41.847000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01457, (ref-fp64): 0.01495 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8483498Z E0601 05:19:41.847000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01305, (ref-fp64): 0.01404 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8489038Z E0601 05:19:41.848000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01020, (ref-fp64): 0.01084 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8494192Z E0601 05:19:41.849000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00909, (ref-fp64): 0.00969 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8499649Z E0601 05:19:41.849000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00985, (ref-fp64): 0.01045 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8504496Z E0601 05:19:41.850000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01558, (ref-fp64): 0.01439 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8509554Z E0601 05:19:41.850000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01514, (ref-fp64): 0.01351 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8514632Z E0601 05:19:41.851000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01720, (ref-fp64): 0.01564 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8519453Z E0601 05:19:41.851000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01662, (ref-fp64): 0.01511 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8524474Z E0601 05:19:41.852000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01388, (ref-fp64): 0.01443 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8529674Z E0601 05:19:41.852000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01251, (ref-fp64): 0.01312 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8534942Z E0601 05:19:41.853000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01127, (ref-fp64): 0.01085 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8540345Z E0601 05:19:41.853000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00999, (ref-fp64): 0.00937 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8545429Z E0601 05:19:41.854000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00924, (ref-fp64): 0.00968 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8550796Z E0601 05:19:41.854000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01220, (ref-fp64): 0.01551 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8555691Z E0601 05:19:41.855000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01113, (ref-fp64): 0.01656 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8560812Z E0601 05:19:41.855000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01377, (ref-fp64): 0.01561 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8565870Z E0601 05:19:41.856000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01436, (ref-fp64): 0.01587 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8570856Z E0601 05:19:41.856000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01432, (ref-fp64): 0.01462 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8575770Z E0601 05:19:41.857000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01186, (ref-fp64): 0.01291 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8581103Z E0601 05:19:41.857000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00910, (ref-fp64): 0.01151 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8586385Z E0601 05:19:41.858000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.01056 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8591759Z E0601 05:19:41.858000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00991, (ref-fp64): 0.01035 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8596734Z E0601 05:19:41.859000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01392, (ref-fp64): 0.01413 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8601642Z E0601 05:19:41.859000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01430, (ref-fp64): 0.01424 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8607019Z E0601 05:19:41.860000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01359, (ref-fp64): 0.01406 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8611727Z E0601 05:19:41.860000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01374, (ref-fp64): 0.01411 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8616729Z E0601 05:19:41.861000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01475, (ref-fp64): 0.01553 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8621703Z E0601 05:19:41.861000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01151, (ref-fp64): 0.01309 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8627081Z E0601 05:19:41.862000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01058, (ref-fp64): 0.01060 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8632196Z E0601 05:19:41.862000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00920, (ref-fp64): 0.00914 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8637845Z E0601 05:19:41.863000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00937, (ref-fp64): 0.00983 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8642717Z E0601 05:19:41.863000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01368, (ref-fp64): 0.01452 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8647914Z E0601 05:19:41.864000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01349, (ref-fp64): 0.01426 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8652710Z E0601 05:19:41.864000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01522, (ref-fp64): 0.01578 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8657510Z E0601 05:19:41.865000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01334, (ref-fp64): 0.01414 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8662377Z E0601 05:19:41.865000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01246, (ref-fp64): 0.01345 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8667241Z E0601 05:19:41.866000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01095, (ref-fp64): 0.01198 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8672649Z E0601 05:19:41.866000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01026, (ref-fp64): 0.01087 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8678001Z E0601 05:19:41.867000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00819, (ref-fp64): 0.00847 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8683124Z E0601 05:19:41.867000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00938, (ref-fp64): 0.01015 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8688531Z E0601 05:19:41.868000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00961, (ref-fp64): 0.01039 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8693249Z E0601 05:19:41.868000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01246, (ref-fp64): 0.01345 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8698046Z E0601 05:19:41.869000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01190, (ref-fp64): 0.01265 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8702821Z E0601 05:19:41.869000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01129, (ref-fp64): 0.01251 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8707741Z E0601 05:19:41.870000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01163, (ref-fp64): 0.01242 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8712304Z E0601 05:19:41.870000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01283, (ref-fp64): 0.01456 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8717091Z E0601 05:19:41.871000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01266, (ref-fp64): 0.01364 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8721770Z E0601 05:19:41.871000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01295, (ref-fp64): 0.01414 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8726988Z E0601 05:19:41.872000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01219, (ref-fp64): 0.01339 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8732408Z E0601 05:19:41.872000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00758, (ref-fp64): 0.00774 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8737690Z E0601 05:19:41.873000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00767, (ref-fp64): 0.00823 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8742688Z E0601 05:19:41.873000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00864, (ref-fp64): 0.00914 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8747651Z E0601 05:19:41.874000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01276, (ref-fp64): 0.01316 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8752480Z E0601 05:19:41.874000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01243, (ref-fp64): 0.01313 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8757355Z E0601 05:19:41.875000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01603 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8762159Z E0601 05:19:41.875000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01471, (ref-fp64): 0.01563 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8767271Z E0601 05:19:41.876000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01395, (ref-fp64): 0.01564 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8772084Z E0601 05:19:41.876000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01252, (ref-fp64): 0.01363 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8777346Z E0601 05:19:41.877000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00885, (ref-fp64): 0.00886 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8782692Z E0601 05:19:41.877000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00821, (ref-fp64): 0.00884 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8787719Z E0601 05:19:41.878000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00919, (ref-fp64): 0.00978 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8792687Z E0601 05:19:41.878000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01345, (ref-fp64): 0.01527 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8797458Z E0601 05:19:41.879000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01304, (ref-fp64): 0.01519 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8802355Z E0601 05:19:41.879000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01505, (ref-fp64): 0.01658 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8807644Z E0601 05:19:41.880000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01523, (ref-fp64): 0.01671 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8812500Z E0601 05:19:41.880000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01399, (ref-fp64): 0.01601 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8817128Z E0601 05:19:41.881000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01229, (ref-fp64): 0.01359 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8822509Z E0601 05:19:41.881000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00928, (ref-fp64): 0.01039 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8827712Z E0601 05:19:41.882000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00930, (ref-fp64): 0.01026 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8832954Z E0601 05:19:41.882000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00933, (ref-fp64): 0.01010 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8837728Z E0601 05:19:41.883000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01347, (ref-fp64): 0.01510 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8842527Z E0601 05:19:41.883000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01347, (ref-fp64): 0.01470 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8847707Z E0601 05:19:41.884000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01493, (ref-fp64): 0.01637 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8852490Z E0601 05:19:41.884000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01601 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8857154Z E0601 05:19:41.885000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01439, (ref-fp64): 0.01628 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8862010Z E0601 05:19:41.885000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01249, (ref-fp64): 0.01357 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8867261Z E0601 05:19:41.886000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00930, (ref-fp64): 0.01015 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8872673Z E0601 05:19:41.886000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00877, (ref-fp64): 0.00930 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8877917Z E0601 05:19:41.887000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.01038 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8882560Z E0601 05:19:41.887000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01419, (ref-fp64): 0.01673 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8887638Z E0601 05:19:41.888000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01656 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8892457Z E0601 05:19:41.888000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01554, (ref-fp64): 0.01746 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8897240Z E0601 05:19:41.889000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01578, (ref-fp64): 0.01763 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8902252Z E0601 05:19:41.889000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01417, (ref-fp64): 0.01626 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8907287Z E0601 05:19:41.890000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01241, (ref-fp64): 0.01370 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8912285Z E0601 05:19:41.890000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00967, (ref-fp64): 0.01098 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8917942Z E0601 05:19:41.891000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00919, (ref-fp64): 0.01035 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8922923Z E0601 05:19:41.891000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00979, (ref-fp64): 0.01075 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8927977Z E0601 05:19:41.892000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01342, (ref-fp64): 0.01531 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8932779Z E0601 05:19:41.892000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01339, (ref-fp64): 0.01515 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8937654Z E0601 05:19:41.893000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01472, (ref-fp64): 0.01551 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8942509Z E0601 05:19:41.893000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01464, (ref-fp64): 0.01540 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8947359Z E0601 05:19:41.894000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01532, (ref-fp64): 0.01640 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8952177Z E0601 05:19:41.894000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01337, (ref-fp64): 0.01428 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8957552Z E0601 05:19:41.895000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00934, (ref-fp64): 0.01051 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8962785Z E0601 05:19:41.895000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00846, (ref-fp64): 0.00874 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8968329Z E0601 05:19:41.896000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00928, (ref-fp64): 0.00980 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8972992Z E0601 05:19:41.896000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01501, (ref-fp64): 0.01610 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8977879Z E0601 05:19:41.897000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01504, (ref-fp64): 0.01597 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8982733Z E0601 05:19:41.897000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01494, (ref-fp64): 0.01656 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8987695Z E0601 05:19:41.898000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01472, (ref-fp64): 0.01622 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.8992864Z E0601 05:19:41.898000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01525, (ref-fp64): 0.01596 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.8997433Z E0601 05:19:41.899000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01278, (ref-fp64): 0.01356 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9002558Z E0601 05:19:41.899000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01046, (ref-fp64): 0.01122 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9008094Z E0601 05:19:41.900000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00887, (ref-fp64): 0.00950 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9013190Z E0601 05:19:41.900000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00979, (ref-fp64): 0.01026 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9017874Z E0601 05:19:41.901000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01555, (ref-fp64): 0.01605 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9022753Z E0601 05:19:41.901000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01558, (ref-fp64): 0.01608 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9027632Z E0601 05:19:41.902000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01547, (ref-fp64): 0.01641 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9032642Z E0601 05:19:41.902000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01560, (ref-fp64): 0.01655 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9037435Z E0601 05:19:41.903000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01521, (ref-fp64): 0.01598 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9042337Z E0601 05:19:41.903000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01355, (ref-fp64): 0.01429 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9047838Z E0601 05:19:41.904000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01059, (ref-fp64): 0.01078 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9053001Z E0601 05:19:41.904000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00898, (ref-fp64): 0.00945 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9058247Z E0601 05:19:41.905000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00954, (ref-fp64): 0.00994 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9062846Z E0601 05:19:41.905000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01576, (ref-fp64): 0.01606 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9067698Z E0601 05:19:41.906000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01526, (ref-fp64): 0.01561 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9072572Z E0601 05:19:41.906000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01723, (ref-fp64): 0.01741 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9077408Z E0601 05:19:41.907000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01717, (ref-fp64): 0.01762 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9082334Z E0601 05:19:41.907000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01494, (ref-fp64): 0.01573 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9087432Z E0601 05:19:41.908000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01277, (ref-fp64): 0.01348 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9092699Z E0601 05:19:41.908000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01078, (ref-fp64): 0.01104 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9097773Z E0601 05:19:41.909000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00978, (ref-fp64): 0.00993 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9102853Z E0601 05:19:41.909000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00946, (ref-fp64): 0.00994 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9107556Z E0601 05:19:41.910000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01520, (ref-fp64): 0.01468 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9112685Z E0601 05:19:41.910000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01495, (ref-fp64): 0.01449 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9117585Z E0601 05:19:41.911000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01604, (ref-fp64): 0.01659 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9122278Z E0601 05:19:41.911000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01603, (ref-fp64): 0.01653 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9127609Z E0601 05:19:41.912000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01527, (ref-fp64): 0.01625 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9132404Z E0601 05:19:41.912000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01283, (ref-fp64): 0.01361 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9137538Z E0601 05:19:41.913000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00991, (ref-fp64): 0.00986 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9142950Z E0601 05:19:41.913000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00925, (ref-fp64): 0.00956 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9147982Z E0601 05:19:41.914000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00982, (ref-fp64): 0.01035 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9152693Z E0601 05:19:41.914000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01522, (ref-fp64): 0.01578 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9157632Z E0601 05:19:41.915000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01510, (ref-fp64): 0.01554 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9162531Z E0601 05:19:41.915000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01650, (ref-fp64): 0.01738 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9167720Z E0601 05:19:41.916000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01658, (ref-fp64): 0.01756 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9172376Z E0601 05:19:41.916000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01511, (ref-fp64): 0.01621 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9177399Z E0601 05:19:41.917000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01269, (ref-fp64): 0.01385 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9182414Z E0601 05:19:41.917000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01050, (ref-fp64): 0.01069 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9187681Z E0601 05:19:41.918000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00967, (ref-fp64): 0.01008 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9192775Z E0601 05:19:41.918000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00983, (ref-fp64): 0.01050 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9197547Z E0601 05:19:41.919000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01455, (ref-fp64): 0.01467 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9202466Z E0601 05:19:41.919000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01300, (ref-fp64): 0.01389 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9207663Z E0601 05:19:41.920000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01752, (ref-fp64): 0.01790 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9212611Z E0601 05:19:41.920000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01718, (ref-fp64): 0.01774 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9217231Z E0601 05:19:41.921000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01385, (ref-fp64): 0.01499 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9222066Z E0601 05:19:41.921000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01169, (ref-fp64): 0.01267 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9227232Z E0601 05:19:41.922000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00915, (ref-fp64): 0.00929 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9232745Z E0601 05:19:41.922000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.00960 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9237805Z E0601 05:19:41.923000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00906, (ref-fp64): 0.00950 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9242630Z E0601 05:19:41.923000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01532, (ref-fp64): 0.01660 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9247883Z E0601 05:19:41.924000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01496, (ref-fp64): 0.01633 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9252535Z E0601 05:19:41.924000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01649, (ref-fp64): 0.01744 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9257184Z E0601 05:19:41.925000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01629, (ref-fp64): 0.01746 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9262204Z E0601 05:19:41.925000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01520, (ref-fp64): 0.01626 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9266924Z E0601 05:19:41.926000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01300, (ref-fp64): 0.01388 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9272382Z E0601 05:19:41.926000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01022, (ref-fp64): 0.01102 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9277532Z E0601 05:19:41.927000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00938, (ref-fp64): 0.00978 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9282844Z E0601 05:19:41.927000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00963, (ref-fp64): 0.01009 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9287727Z E0601 05:19:41.928000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01461, (ref-fp64): 0.01636 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9292703Z E0601 05:19:41.928000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01463, (ref-fp64): 0.01626 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9297318Z E0601 05:19:41.929000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01662, (ref-fp64): 0.01754 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9302352Z E0601 05:19:41.929000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01641, (ref-fp64): 0.01752 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9307063Z E0601 05:19:41.930000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01583, (ref-fp64): 0.01673 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9312131Z E0601 05:19:41.930000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01353, (ref-fp64): 0.01436 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9317330Z E0601 05:19:41.931000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00960, (ref-fp64): 0.01052 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9322512Z E0601 05:19:41.931000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00902, (ref-fp64): 0.00950 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9327899Z E0601 05:19:41.932000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01005, (ref-fp64): 0.01055 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9332539Z E0601 05:19:41.932000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01624, (ref-fp64): 0.01689 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9337508Z E0601 05:19:41.933000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01576, (ref-fp64): 0.01627 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9342137Z E0601 05:19:41.933000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01677, (ref-fp64): 0.01806 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9346980Z E0601 05:19:41.934000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01673, (ref-fp64): 0.01758 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9352122Z E0601 05:19:41.934000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01560, (ref-fp64): 0.01668 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9357243Z E0601 05:19:41.935000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01361, (ref-fp64): 0.01436 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9362305Z E0601 05:19:41.935000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01092, (ref-fp64): 0.01133 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9367926Z E0601 05:19:41.936000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.01011 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9372912Z E0601 05:19:41.936000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01037, (ref-fp64): 0.01088 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9377488Z E0601 05:19:41.937000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01556, (ref-fp64): 0.01611 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9382382Z E0601 05:19:41.937000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01574, (ref-fp64): 0.01589 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9386984Z E0601 05:19:41.938000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01745, (ref-fp64): 0.01713 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9391896Z E0601 05:19:41.938000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01766, (ref-fp64): 0.01713 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9398672Z E0601 05:19:41.939000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01601, (ref-fp64): 0.01673 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9403340Z E0601 05:19:41.939000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01387, (ref-fp64): 0.01456 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9408817Z E0601 05:19:41.940000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01043, (ref-fp64): 0.01095 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9413884Z E0601 05:19:41.940000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01004, (ref-fp64): 0.01013 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9418978Z E0601 05:19:41.941000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01050, (ref-fp64): 0.01094 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9423701Z E0601 05:19:41.941000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01539, (ref-fp64): 0.01618 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9428433Z E0601 05:19:41.942000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01551, (ref-fp64): 0.01644 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9433465Z E0601 05:19:41.942000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01717, (ref-fp64): 0.01726 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9437999Z E0601 05:19:41.943000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01721, (ref-fp64): 0.01732 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9442977Z E0601 05:19:41.943000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01637, (ref-fp64): 0.01728 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9448102Z E0601 05:19:41.944000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01430, (ref-fp64): 0.01525 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9453267Z E0601 05:19:41.944000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01082, (ref-fp64): 0.01120 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9458415Z E0601 05:19:41.945000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.01006 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9463554Z E0601 05:19:41.945000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01052, (ref-fp64): 0.01100 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9468198Z E0601 05:19:41.946000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01796, (ref-fp64): 0.01871 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9473062Z E0601 05:19:41.946000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01779, (ref-fp64): 0.01854 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9477823Z E0601 05:19:41.947000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01860, (ref-fp64): 0.01864 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9482620Z E0601 05:19:41.947000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01861, (ref-fp64): 0.01856 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9487712Z E0601 05:19:41.948000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01656, (ref-fp64): 0.01721 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9492483Z E0601 05:19:41.948000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01461, (ref-fp64): 0.01519 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9498306Z E0601 05:19:41.949000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01233, (ref-fp64): 0.01292 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9502917Z E0601 05:19:41.949000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01029, (ref-fp64): 0.01043 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9507837Z E0601 05:19:41.950000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01073, (ref-fp64): 0.01115 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9512831Z E0601 05:19:41.950000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01737, (ref-fp64): 0.01762 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9518202Z E0601 05:19:41.951000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01729, (ref-fp64): 0.01729 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9522700Z E0601 05:19:41.951000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01750, (ref-fp64): 0.01732 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9527774Z E0601 05:19:41.952000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01734, (ref-fp64): 0.01745 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9532575Z E0601 05:19:41.952000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01548, (ref-fp64): 0.01609 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9537374Z E0601 05:19:41.953000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01337, (ref-fp64): 0.01417 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9542531Z E0601 05:19:41.953000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01230, (ref-fp64): 0.01236 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9547716Z E0601 05:19:41.954000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01038, (ref-fp64): 0.01058 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9553054Z E0601 05:19:41.954000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00985, (ref-fp64): 0.01029 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9558134Z E0601 05:19:41.955000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01580, (ref-fp64): 0.01605 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9562880Z E0601 05:19:41.955000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01561, (ref-fp64): 0.01580 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9567924Z E0601 05:19:41.956000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01697, (ref-fp64): 0.01626 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9572689Z E0601 05:19:41.956000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01658, (ref-fp64): 0.01576 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9577411Z E0601 05:19:41.957000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01549, (ref-fp64): 0.01636 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9582208Z E0601 05:19:41.957000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01303, (ref-fp64): 0.01407 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9587451Z E0601 05:19:41.958000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01088, (ref-fp64): 0.01098 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9592873Z E0601 05:19:41.958000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00967, (ref-fp64): 0.00983 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9598026Z E0601 05:19:41.959000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00978, (ref-fp64): 0.01034 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9602874Z E0601 05:19:41.959000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01686, (ref-fp64): 0.01792 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9608003Z E0601 05:19:41.960000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01648, (ref-fp64): 0.01732 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9612742Z E0601 05:19:41.960000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01816, (ref-fp64): 0.01813 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9617401Z E0601 05:19:41.961000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01804, (ref-fp64): 0.01795 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9622477Z E0601 05:19:41.961000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01466, (ref-fp64): 0.01586 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9627032Z E0601 05:19:41.962000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01303, (ref-fp64): 0.01376 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9632668Z E0601 05:19:41.962000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01156, (ref-fp64): 0.01203 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9637601Z E0601 05:19:41.963000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01063, (ref-fp64): 0.01081 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9642785Z E0601 05:19:41.963000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00969, (ref-fp64): 0.01030 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9647978Z E0601 05:19:41.964000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01422, (ref-fp64): 0.01541 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9652806Z E0601 05:19:41.964000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01441, (ref-fp64): 0.01532 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9657737Z E0601 05:19:41.965000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01496, (ref-fp64): 0.01563 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9662534Z E0601 05:19:41.965000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01568 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9667329Z E0601 05:19:41.966000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01523, (ref-fp64): 0.01591 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9672449Z E0601 05:19:41.966000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01301, (ref-fp64): 0.01373 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9677615Z E0601 05:19:41.967000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00966, (ref-fp64): 0.01043 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9682861Z E0601 05:19:41.967000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00866, (ref-fp64): 0.00922 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9688238Z E0601 05:19:41.968000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00945, (ref-fp64): 0.00987 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9692957Z E0601 05:19:41.968000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01504, (ref-fp64): 0.01604 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9697859Z E0601 05:19:41.969000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01460, (ref-fp64): 0.01564 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9702896Z E0601 05:19:41.969000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01484, (ref-fp64): 0.01554 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9707780Z E0601 05:19:41.970000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01480, (ref-fp64): 0.01520 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9712511Z E0601 05:19:41.970000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01371, (ref-fp64): 0.01527 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9717357Z E0601 05:19:41.971000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01205, (ref-fp64): 0.01292 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9724143Z E0601 05:19:41.971000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00972, (ref-fp64): 0.01020 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9729808Z E0601 05:19:41.972000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00874, (ref-fp64): 0.00916 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9735025Z E0601 05:19:41.973000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00865, (ref-fp64): 0.00914 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9739635Z E0601 05:19:41.973000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01476, (ref-fp64): 0.01601 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9744515Z E0601 05:19:41.974000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01424, (ref-fp64): 0.01562 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9749291Z E0601 05:19:41.974000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01519, (ref-fp64): 0.01554 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9754269Z E0601 05:19:41.975000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01463, (ref-fp64): 0.01493 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9759007Z E0601 05:19:41.975000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01415, (ref-fp64): 0.01547 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9763934Z E0601 05:19:41.975000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01250, (ref-fp64): 0.01340 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9769179Z E0601 05:19:41.976000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01019, (ref-fp64): 0.01100 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9774766Z E0601 05:19:41.977000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00858, (ref-fp64): 0.00905 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9779615Z E0601 05:19:41.977000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00874, (ref-fp64): 0.00938 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9784590Z E0601 05:19:41.978000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01447, (ref-fp64): 0.01544 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9789239Z E0601 05:19:41.978000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01455, (ref-fp64): 0.01528 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9794123Z E0601 05:19:41.979000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01381, (ref-fp64): 0.01603 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9798741Z E0601 05:19:41.979000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01362, (ref-fp64): 0.01577 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9803730Z E0601 05:19:41.979000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01357, (ref-fp64): 0.01512 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9808803Z E0601 05:19:41.980000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01238, (ref-fp64): 0.01342 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9813955Z E0601 05:19:41.980000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00983, (ref-fp64): 0.01039 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9819232Z E0601 05:19:41.981000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00783, (ref-fp64): 0.00874 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9824341Z E0601 05:19:41.982000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00859, (ref-fp64): 0.00932 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9829187Z E0601 05:19:41.982000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01383, (ref-fp64): 0.01561 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9834208Z E0601 05:19:41.983000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01371, (ref-fp64): 0.01550 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9838769Z E0601 05:19:41.983000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01464, (ref-fp64): 0.01580 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9843804Z E0601 05:19:41.983000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01585 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9848802Z E0601 05:19:41.984000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01401, (ref-fp64): 0.01529 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9853680Z E0601 05:19:41.984000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01230, (ref-fp64): 0.01337 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9859068Z E0601 05:19:41.985000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00945, (ref-fp64): 0.01054 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9864137Z E0601 05:19:41.986000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00803, (ref-fp64): 0.00890 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9869213Z E0601 05:19:41.986000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00902, (ref-fp64): 0.00970 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9874327Z E0601 05:19:41.987000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01435, (ref-fp64): 0.01478 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9878840Z E0601 05:19:41.987000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01431, (ref-fp64): 0.01431 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9883920Z E0601 05:19:41.987000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01417, (ref-fp64): 0.01487 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9888829Z E0601 05:19:41.988000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01413, (ref-fp64): 0.01481 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9893678Z E0601 05:19:41.988000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01410, (ref-fp64): 0.01507 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9898537Z E0601 05:19:41.989000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01264, (ref-fp64): 0.01372 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9903628Z E0601 05:19:41.989000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00940, (ref-fp64): 0.00984 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9908867Z E0601 05:19:41.990000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00730, (ref-fp64): 0.00796 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9914278Z E0601 05:19:41.991000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00812, (ref-fp64): 0.00874 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9918826Z E0601 05:19:41.991000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01354, (ref-fp64): 0.01377 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9923783Z E0601 05:19:41.991000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01378, (ref-fp64): 0.01391 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9928717Z E0601 05:19:41.992000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01498, (ref-fp64): 0.01624 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9933591Z E0601 05:19:41.992000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01486, (ref-fp64): 0.01613 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9938367Z E0601 05:19:41.993000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01412, (ref-fp64): 0.01546 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9943304Z E0601 05:19:41.993000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01249, (ref-fp64): 0.01376 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9948345Z E0601 05:19:41.994000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.00939 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9953897Z E0601 05:19:41.994000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00732, (ref-fp64): 0.00821 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9958825Z E0601 05:19:41.995000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00878, (ref-fp64): 0.00953 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9963861Z E0601 05:19:41.995000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01591, (ref-fp64): 0.01746 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9968816Z E0601 05:19:41.996000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01575, (ref-fp64): 0.01705 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9973526Z E0601 05:19:41.996000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01522, (ref-fp64): 0.01575 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9978384Z E0601 05:19:41.997000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01498, (ref-fp64): 0.01566 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9983215Z E0601 05:19:41.997000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01394, (ref-fp64): 0.01519 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9988104Z E0601 05:19:41.998000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01225, (ref-fp64): 0.01365 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:41.9993201Z E0601 05:19:41.998000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01020, (ref-fp64): 0.01121 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:41.9998666Z E0601 05:19:41.999000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00826, (ref-fp64): 0.00923 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0004041Z E0601 05:19:41.999000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00844, (ref-fp64): 0.00927 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0008934Z E0601 05:19:42.000000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01542, (ref-fp64): 0.01815 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0013824Z E0601 05:19:42.000000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01498, (ref-fp64): 0.01777 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0018567Z E0601 05:19:42.001000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01518, (ref-fp64): 0.01663 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0023438Z E0601 05:19:42.001000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01504, (ref-fp64): 0.01652 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0028273Z E0601 05:19:42.002000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01397, (ref-fp64): 0.01560 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0033296Z E0601 05:19:42.002000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01182, (ref-fp64): 0.01318 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0038414Z E0601 05:19:42.003000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01015, (ref-fp64): 0.01173 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0043774Z E0601 05:19:42.003000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00949, (ref-fp64): 0.01005 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0049127Z E0601 05:19:42.004000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00943, (ref-fp64): 0.01020 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0054105Z E0601 05:19:42.005000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01218, (ref-fp64): 0.01540 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0058661Z E0601 05:19:42.005000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01201, (ref-fp64): 0.01527 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0063678Z E0601 05:19:42.005000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01406, (ref-fp64): 0.01628 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0068489Z E0601 05:19:42.006000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01393, (ref-fp64): 0.01603 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0073451Z E0601 05:19:42.006000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01397, (ref-fp64): 0.01562 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0078102Z E0601 05:19:42.007000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01149, (ref-fp64): 0.01283 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0083353Z E0601 05:19:42.007000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00882, (ref-fp64): 0.01072 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0088795Z E0601 05:19:42.008000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00783, (ref-fp64): 0.00879 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0094196Z E0601 05:19:42.009000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00887, (ref-fp64): 0.00958 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0098906Z E0601 05:19:42.009000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01276, (ref-fp64): 0.01467 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0103955Z E0601 05:19:42.010000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01292, (ref-fp64): 0.01445 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0108965Z E0601 05:19:42.010000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01507, (ref-fp64): 0.01606 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0113798Z E0601 05:19:42.010000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01504, (ref-fp64): 0.01598 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0118690Z E0601 05:19:42.011000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01560 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0123635Z E0601 05:19:42.011000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01289, (ref-fp64): 0.01337 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0129005Z E0601 05:19:42.012000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00874, (ref-fp64): 0.01006 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0134440Z E0601 05:19:42.013000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00855, (ref-fp64): 0.00885 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0139716Z E0601 05:19:42.013000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00943, (ref-fp64): 0.00982 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0144415Z E0601 05:19:42.014000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01384, (ref-fp64): 0.01437 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0149324Z E0601 05:19:42.014000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01387 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0154266Z E0601 05:19:42.015000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01668, (ref-fp64): 0.01667 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0158854Z E0601 05:19:42.015000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01682, (ref-fp64): 0.01634 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0164033Z E0601 05:19:42.016000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01564 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0168916Z E0601 05:19:42.016000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01269, (ref-fp64): 0.01380 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0174203Z E0601 05:19:42.017000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00935, (ref-fp64): 0.00952 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0179725Z E0601 05:19:42.017000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00958, (ref-fp64): 0.00929 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0184802Z E0601 05:19:42.018000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00977, (ref-fp64): 0.01011 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0189683Z E0601 05:19:42.018000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01380, (ref-fp64): 0.01443 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0194663Z E0601 05:19:42.019000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01385, (ref-fp64): 0.01417 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0199248Z E0601 05:19:42.019000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01351, (ref-fp64): 0.01402 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0204573Z E0601 05:19:42.020000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01329, (ref-fp64): 0.01367 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0209305Z E0601 05:19:42.020000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01408, (ref-fp64): 0.01530 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0214376Z E0601 05:19:42.021000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01224, (ref-fp64): 0.01332 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0219600Z E0601 05:19:42.021000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00932, (ref-fp64): 0.00953 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0225012Z E0601 05:19:42.022000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00756, (ref-fp64): 0.00771 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0230033Z E0601 05:19:42.022000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00863, (ref-fp64): 0.00907 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0235043Z E0601 05:19:42.023000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01520 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0239683Z E0601 05:19:42.023000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01347, (ref-fp64): 0.01487 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0244741Z E0601 05:19:42.024000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01503, (ref-fp64): 0.01727 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0249651Z E0601 05:19:42.024000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01486, (ref-fp64): 0.01700 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0254444Z E0601 05:19:42.025000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01376, (ref-fp64): 0.01494 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0259316Z E0601 05:19:42.025000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01198, (ref-fp64): 0.01311 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0264616Z E0601 05:19:42.026000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00924, (ref-fp64): 0.01000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0269901Z E0601 05:19:42.026000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00826, (ref-fp64): 0.00900 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0275305Z E0601 05:19:42.027000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00906, (ref-fp64): 0.00948 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0279910Z E0601 05:19:42.027000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01321, (ref-fp64): 0.01407 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0285200Z E0601 05:19:42.028000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01341, (ref-fp64): 0.01432 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0289889Z E0601 05:19:42.028000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01305, (ref-fp64): 0.01337 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0294950Z E0601 05:19:42.029000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01282, (ref-fp64): 0.01327 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0299482Z E0601 05:19:42.029000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01172, (ref-fp64): 0.01105 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0304328Z E0601 05:19:42.030000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01076, (ref-fp64): 0.01017 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0309546Z E0601 05:19:42.030000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00860, (ref-fp64): 0.00926 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0322330Z E0601 05:19:42.031000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00659, (ref-fp64): 0.00687 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0329202Z E0601 05:19:42.032000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00782, (ref-fp64): 0.00821 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0340464Z E0601 05:19:42.033000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00800, (ref-fp64): 0.00864 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0345149Z E0601 05:19:42.034000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01172, (ref-fp64): 0.01105 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0349884Z E0601 05:19:42.034000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01070, (ref-fp64): 0.01040 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0354738Z E0601 05:19:42.035000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01071, (ref-fp64): 0.01219 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0359489Z E0601 05:19:42.035000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01072, (ref-fp64): 0.01210 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0364346Z E0601 05:19:42.036000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01126, (ref-fp64): 0.01261 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0369488Z E0601 05:19:42.036000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01097, (ref-fp64): 0.01224 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0374043Z E0601 05:19:42.037000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01165, (ref-fp64): 0.01093 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0378913Z E0601 05:19:42.037000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01059, (ref-fp64): 0.01003 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0385603Z E0601 05:19:42.038000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00590, (ref-fp64): 0.00739 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0398347Z E0601 05:19:42.039000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00442, (ref-fp64): 0.00597 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0405130Z E0601 05:19:42.040000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00678, (ref-fp64): 0.00804 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0409960Z E0601 05:19:42.040000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00874, (ref-fp64): 0.01037 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0414749Z E0601 05:19:42.041000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00873, (ref-fp64): 0.01032 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0419463Z E0601 05:19:42.041000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01299, (ref-fp64): 0.01424 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0424124Z E0601 05:19:42.042000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01186, (ref-fp64): 0.01329 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0428917Z E0601 05:19:42.042000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01162, (ref-fp64): 0.00921 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0433596Z E0601 05:19:42.042000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00998, (ref-fp64): 0.00914 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0440342Z E0601 05:19:42.043000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00486, (ref-fp64): 0.00629 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.0453241Z E0601 05:19:42.044000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00316, (ref-fp64): 0.00510 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:19:42.0459714Z E0601 05:19:42.045000 139951583220352 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00671, (ref-fp64): 0.00786 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:19:42.1444088Z pass 2024-06-01T05:19:42.1804348Z TIMING: entire_frame_compile:225.72313 code_gen:55.49127 inductor_compile:122.58408 backend_compile:196.59275 2024-06-01T05:19:42.1805972Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:41495 | FakeTensorMode.__torch_dispatch__:259016 | attempt fast:4445 | fast is_contiguous:4445 | ProxyTorchDispatchMode.__torch_dispatch__:53818 2024-06-01T05:19:42.1807402Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-06-01T05:19:56.9000244Z 2024-06-01T05:19:58.2591812Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:19:58.2592313Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:19:58.2592769Z cuda train resnet18 2024-06-01T05:20:20.8356804Z E0601 05:20:20.834000 139753756361344 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00167, (ref-fp64): 0.00174 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:20:20.8774267Z pass 2024-06-01T05:20:20.8851088Z TIMING: entire_frame_compile:18.94738 code_gen:5.84965 inductor_compile:11.56214 backend_compile:17.44695 2024-06-01T05:20:20.8853088Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:2577 | FakeTensorMode.__torch_dispatch__:17996 | attempt fast:590 | fast is_contiguous:590 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-06-01T05:20:20.8854492Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-06-01T05:20:25.2782543Z 2024-06-01T05:20:27.2349280Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:20:27.2349920Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:20:27.2351026Z cuda train resnet50 2024-06-01T05:21:16.5243483Z E0601 05:21:16.523000 139974019732096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00191, (ref-fp64): 0.00208 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:21:16.5561138Z E0601 05:21:16.555000 139974019732096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00000, (ref-fp64): 0.00007 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:21:16.6290483Z pass 2024-06-01T05:21:16.6469481Z TIMING: entire_frame_compile:41.17104 code_gen:9.17765 inductor_compile:22.94011 backend_compile:37.73951 2024-06-01T05:21:16.6473421Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:6682 | FakeTensorMode.__torch_dispatch__:46310 | attempt fast:1521 | fast is_contiguous:1521 | ProxyTorchDispatchMode.__torch_dispatch__:11223 2024-06-01T05:21:16.6474854Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-06-01T05:21:22.1701972Z 2024-06-01T05:21:24.1920344Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:21:24.1920909Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:21:24.1924166Z cuda train resnet50_quantized_qat 2024-06-01T05:21:24.1924728Z Traceback (most recent call last): 2024-06-01T05:21:24.1925926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model 2024-06-01T05:21:24.1926737Z self.model_iter_fn(model, example_inputs) 2024-06-01T05:21:24.1927837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in forward_and_backward_pass 2024-06-01T05:21:24.1928753Z pred = mod(*cloned_inputs) 2024-06-01T05:21:24.1930040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 737, in call_wrapped 2024-06-01T05:21:24.1931086Z return self._wrapped_call(self, *args, **kwargs) 2024-06-01T05:21:24.1932290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__ 2024-06-01T05:21:24.1933405Z raise e 2024-06-01T05:21:24.1934442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__ 2024-06-01T05:21:24.1935444Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-01T05:21:24.1936617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1540, in _wrapped_call_impl 2024-06-01T05:21:24.1937544Z return self._call_impl(*args, **kwargs) 2024-06-01T05:21:24.1938504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:21:24.1939374Z return forward_call(*args, **kwargs) 2024-06-01T05:21:24.1941244Z File ".3", line 167, in forward 2024-06-01T05:21:24.1941917Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-06-01T05:21:24.1943104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1540, in _wrapped_call_impl 2024-06-01T05:21:24.1944305Z return self._call_impl(*args, **kwargs) 2024-06-01T05:21:24.1945248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:21:24.1946106Z return forward_call(*args, **kwargs) 2024-06-01T05:21:24.1947093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward 2024-06-01T05:21:24.1948028Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-01T05:21:24.1948603Z RuntimeError: expected scalar type Float but found Half 2024-06-01T05:21:24.1948992Z 2024-06-01T05:21:24.1949297Z The above exception was the direct cause of the following exception: 2024-06-01T05:21:24.1949762Z 2024-06-01T05:21:24.1949911Z Traceback (most recent call last): 2024-06-01T05:21:24.1950978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T05:21:24.1951673Z ) = runner.load_model( 2024-06-01T05:21:24.1952481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model 2024-06-01T05:21:24.1953323Z self.validate_model(model, example_inputs) 2024-06-01T05:21:24.1954140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model 2024-06-01T05:21:24.1954954Z raise RuntimeError("Eager run failed") from e 2024-06-01T05:21:24.1955436Z RuntimeError: Eager run failed 2024-06-01T05:21:24.1955709Z 2024-06-01T05:21:24.1955827Z eager_fail_to_run 2024-06-01T05:21:27.4905828Z 2024-06-01T05:21:29.5519280Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:21:29.5519999Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:21:29.5520669Z cuda train resnext50_32x4d 2024-06-01T05:22:19.0736601Z W0601 05:22:19.072000 140049406337664 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:23:02.7007929Z E0601 05:23:02.699000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 37004721939964059648.00000, (ref-fp64): 37004721939964059648.00000 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7011832Z E0601 05:23:02.700000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 36746346425896157184.00000, (ref-fp64): 36746346425896157184.00000 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7016751Z E0601 05:23:02.701000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1080436394639282560.00000, (ref-fp64): 1080436394639282560.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7020710Z E0601 05:23:02.701000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2391393025224170496.00000, (ref-fp64): 2391393025224170496.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7024932Z E0601 05:23:02.702000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 24975794021394844.00000, (ref-fp64): 24975794021394844.00000 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7038466Z E0601 05:23:02.703000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2071331881536352.50000, (ref-fp64): 2071331881536352.50000 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7042767Z E0601 05:23:02.703000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 794754405119088640.00000, (ref-fp64): 794754405119088640.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7046998Z E0601 05:23:02.704000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1693350396942251008.00000, (ref-fp64): 1693350396942251008.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7051277Z E0601 05:23:02.704000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 602331247833134464.00000, (ref-fp64): 602331247833134464.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7055349Z E0601 05:23:02.705000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1943391913961969920.00000, (ref-fp64): 1943391913961969920.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7059594Z E0601 05:23:02.705000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 307113351942385600.00000, (ref-fp64): 307113351942385600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7063830Z E0601 05:23:02.705000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 857761852659111808.00000, (ref-fp64): 857761852659111808.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7068009Z E0601 05:23:02.706000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 490743043255152448.00000, (ref-fp64): 490743043255152448.00000 and shape=torch.Size([128, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7072437Z E0601 05:23:02.706000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 655276351144342144.00000, (ref-fp64): 655276351144342144.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7077276Z E0601 05:23:02.707000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 199646472650708256.00000, (ref-fp64): 199646472650708256.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7081461Z E0601 05:23:02.707000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 297824915587809088.00000, (ref-fp64): 297824915587809088.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7086036Z E0601 05:23:02.708000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 307113351942385600.00000, (ref-fp64): 307113351942385600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7090490Z E0601 05:23:02.708000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1181863427270611200.00000, (ref-fp64): 1181863427270611200.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7094545Z E0601 05:23:02.709000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 443324539170970304.00000, (ref-fp64): 443324539170970304.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7098708Z E0601 05:23:02.709000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2081065677546305024.00000, (ref-fp64): 2081065677546305024.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7102715Z E0601 05:23:02.709000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 397735078670651584.00000, (ref-fp64): 397735078670651584.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7106931Z E0601 05:23:02.710000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1917868354062398720.00000, (ref-fp64): 1917868354062398720.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7111184Z E0601 05:23:02.710000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 144388017049485792.00000, (ref-fp64): 144388017049485792.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7115474Z E0601 05:23:02.711000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1529807768992260864.00000, (ref-fp64): 1529807768992260864.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7119789Z E0601 05:23:02.711000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 322000321946047424.00000, (ref-fp64): 322000321946047424.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7124205Z E0601 05:23:02.711000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 822760044927529984.00000, (ref-fp64): 822760044927529984.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7128797Z E0601 05:23:02.712000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 221943892502712224.00000, (ref-fp64): 221943892502712224.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7132807Z E0601 05:23:02.712000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 229667840425162432.00000, (ref-fp64): 229667840425162432.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7136935Z E0601 05:23:02.713000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3693263356514978304.00000, (ref-fp64): 3693263356514978304.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7141063Z E0601 05:23:02.713000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 198065659175187168.00000, (ref-fp64): 198065659175187168.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7145322Z E0601 05:23:02.714000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2779137313339571200.00000, (ref-fp64): 2779137313339571200.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7149534Z E0601 05:23:02.714000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 68545425084836776.00000, (ref-fp64): 68545425084836776.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7153870Z E0601 05:23:02.714000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2308003723667874816.00000, (ref-fp64): 2308003723667874816.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7158379Z E0601 05:23:02.715000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 313442844223128448.00000, (ref-fp64): 313442844223128448.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7162634Z E0601 05:23:02.715000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1038780123696505472.00000, (ref-fp64): 1038780123696505472.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7167353Z E0601 05:23:02.716000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 300870937660851264.00000, (ref-fp64): 300870937660851264.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7171303Z E0601 05:23:02.716000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58465902246591752.00000, (ref-fp64): 58465902246591760.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7175413Z E0601 05:23:02.717000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2158304688806577152.00000, (ref-fp64): 2158304688806577152.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7179534Z E0601 05:23:02.717000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 18080039714265164.00000, (ref-fp64): 18080039714265164.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7183773Z E0601 05:23:02.717000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1919046935363044352.00000, (ref-fp64): 1919046935363044352.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7187989Z E0601 05:23:02.718000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3978533496692100.50000, (ref-fp64): 3978533496692101.50000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7192244Z E0601 05:23:02.718000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 851251431773208704.00000, (ref-fp64): 851251431773208704.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7197104Z E0601 05:23:02.719000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 184407218608400928.00000, (ref-fp64): 184407218608400928.00000 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7201228Z E0601 05:23:02.719000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 243686482646839776.00000, (ref-fp64): 243686482646839776.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7205887Z E0601 05:23:02.720000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 89718028331538528.00000, (ref-fp64): 89718028331538528.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7210461Z E0601 05:23:02.720000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 11985594379755916.00000, (ref-fp64): 11985594379755916.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7214574Z E0601 05:23:02.721000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3978533496692100.50000, (ref-fp64): 3978533496692101.50000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7218766Z E0601 05:23:02.721000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 144621713400583712.00000, (ref-fp64): 144621713400583712.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7222891Z E0601 05:23:02.721000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 6046748783186963.00000, (ref-fp64): 6046748783186964.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7227062Z E0601 05:23:02.722000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3367463887796313088.00000, (ref-fp64): 3367463887796313088.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7231294Z E0601 05:23:02.722000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4451585216033064.50000, (ref-fp64): 4451585216033065.50000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7235510Z E0601 05:23:02.723000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1491942169235417856.00000, (ref-fp64): 1491942169235417856.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7239563Z E0601 05:23:02.723000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1010410556368366.25000, (ref-fp64): 1010410556368366.87500 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7243791Z E0601 05:23:02.723000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1539193095382635264.00000, (ref-fp64): 1539193095382635264.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7248538Z E0601 05:23:02.724000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 139664526067608800.00000, (ref-fp64): 139664526067608800.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7252615Z E0601 05:23:02.724000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 574000937150627392.00000, (ref-fp64): 574000937150627392.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7257804Z E0601 05:23:02.725000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 151365062464393792.00000, (ref-fp64): 151365062464393792.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7261031Z E0601 05:23:02.725000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1472706805780054.00000, (ref-fp64): 1472706805780054.50000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7265354Z E0601 05:23:02.726000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3329311946368118784.00000, (ref-fp64): 3329311946368118784.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7269429Z E0601 05:23:02.726000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 677497429327720.12500, (ref-fp64): 677497429327720.87500 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7273807Z E0601 05:23:02.726000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2722597063803414528.00000, (ref-fp64): 2722597063803414528.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7277889Z E0601 05:23:02.727000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 183921100696933.28125, (ref-fp64): 183921100696933.81250 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7282138Z E0601 05:23:02.727000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2198055149354711040.00000, (ref-fp64): 2198055149354711040.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7287036Z E0601 05:23:02.728000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 166237407580071776.00000, (ref-fp64): 166237407580071776.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7291126Z E0601 05:23:02.728000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 522012777259312064.00000, (ref-fp64): 522012777259312064.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7295606Z E0601 05:23:02.729000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 200047070964461408.00000, (ref-fp64): 200047070964461408.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7299690Z E0601 05:23:02.729000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 304872881572424.56250, (ref-fp64): 304872881572425.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7303910Z E0601 05:23:02.729000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2497688825515727872.00000, (ref-fp64): 2497688825515727872.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7307978Z E0601 05:23:02.730000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 213692290187449.21875, (ref-fp64): 213692290187450.31250 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7312320Z E0601 05:23:02.730000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2102741266366877696.00000, (ref-fp64): 2102741266366877696.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7316415Z E0601 05:23:02.731000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 43229586855358.65625, (ref-fp64): 43229586855359.50781 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7320552Z E0601 05:23:02.731000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1557061734323560192.00000, (ref-fp64): 1557061734323560192.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7325457Z E0601 05:23:02.732000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 228817310693161312.00000, (ref-fp64): 228817310693161312.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7329632Z E0601 05:23:02.732000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 632887190751524480.00000, (ref-fp64): 632887190751524480.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7333954Z E0601 05:23:02.732000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 158302713278241216.00000, (ref-fp64): 158302713278241216.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7338068Z E0601 05:23:02.733000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 22471290464040.28906, (ref-fp64): 22471290464041.24219 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7342296Z E0601 05:23:02.733000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1008565131207630208.00000, (ref-fp64): 1008565131207630208.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7346368Z E0601 05:23:02.734000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 6251494263396.18066, (ref-fp64): 6251494263397.01074 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7350723Z E0601 05:23:02.734000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1058424403032925824.00000, (ref-fp64): 1058424403032925824.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7354863Z E0601 05:23:02.735000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 968222367541.99707, (ref-fp64): 968222367542.38953 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7359188Z E0601 05:23:02.735000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 494376298606308096.00000, (ref-fp64): 494376298606308096.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7363582Z E0601 05:23:02.735000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58192019844584008.00000, (ref-fp64): 58192019844584008.00000 and shape=torch.Size([512, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7368511Z E0601 05:23:02.736000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 92426079555688496.00000, (ref-fp64): 92426079555688496.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7373005Z E0601 05:23:02.736000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 33538693736376796.00000, (ref-fp64): 33538693736376796.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7377613Z E0601 05:23:02.737000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2353690581278669.00000, (ref-fp64): 2353690581278669.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7381607Z E0601 05:23:02.737000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 968222367541.99707, (ref-fp64): 968222367542.38953 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7385851Z E0601 05:23:02.738000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 53089400135336416.00000, (ref-fp64): 53089400135336416.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7390210Z E0601 05:23:02.738000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1491203056359.44604, (ref-fp64): 1491203056359.63037 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7394341Z E0601 05:23:02.739000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2149823010016831744.00000, (ref-fp64): 2149823010016831744.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7398415Z E0601 05:23:02.739000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 665305150782.55579, (ref-fp64): 665305150783.04065 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7402678Z E0601 05:23:02.739000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1309722893007537152.00000, (ref-fp64): 1309722893007537152.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7407104Z E0601 05:23:02.740000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 104931357763.77969, (ref-fp64): 104931357764.16368 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7411161Z E0601 05:23:02.740000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 634470892746599808.00000, (ref-fp64): 634470892746599808.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7415806Z E0601 05:23:02.741000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 63738809478936456.00000, (ref-fp64): 63738809478936456.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7420470Z E0601 05:23:02.741000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 227711746360184288.00000, (ref-fp64): 227711746360184288.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7425013Z E0601 05:23:02.742000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 56884121822411568.00000, (ref-fp64): 56884121822411568.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7429034Z E0601 05:23:02.742000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 116756741058.48878, (ref-fp64): 116756741058.77849 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7433342Z E0601 05:23:02.742000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1682167624979401472.00000, (ref-fp64): 1682167624979401472.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7437504Z E0601 05:23:02.743000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 51065005951.00156, (ref-fp64): 51065005951.63967 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7441689Z E0601 05:23:02.743000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1273156281389955328.00000, (ref-fp64): 1273156281389955328.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7446192Z E0601 05:23:02.744000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 6727927213.71856, (ref-fp64): 6727927214.12361 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7450135Z E0601 05:23:02.744000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 756402576371474048.00000, (ref-fp64): 756402576371474048.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7454702Z E0601 05:23:02.745000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58679691619189656.00000, (ref-fp64): 58679691619189656.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7459605Z E0601 05:23:02.745000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 236708042528294144.00000, (ref-fp64): 236708042528294144.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7464187Z E0601 05:23:02.745000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 54049270738605104.00000, (ref-fp64): 54049270738605104.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7468010Z E0601 05:23:02.746000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 5877575925.96700, (ref-fp64): 5877575926.27232 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7472539Z E0601 05:23:02.746000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1530254061752436480.00000, (ref-fp64): 1530254061752436480.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7476508Z E0601 05:23:02.747000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 2215092023.41601, (ref-fp64): 2215092024.02404 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7480792Z E0601 05:23:02.747000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1440625416364438784.00000, (ref-fp64): 1440625416364438784.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7484978Z E0601 05:23:02.748000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 313575431.88041, (ref-fp64): 313575432.32671 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7489270Z E0601 05:23:02.748000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 944048435654065664.00000, (ref-fp64): 944048435654065664.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7493849Z E0601 05:23:02.748000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 52994548550202128.00000, (ref-fp64): 52994548550202128.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7506822Z E0601 05:23:02.749000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 211774121547048704.00000, (ref-fp64): 211774121547048704.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7509890Z E0601 05:23:02.749000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 57676302636389480.00000, (ref-fp64): 57676302636389480.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7512703Z E0601 05:23:02.750000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 292582225.01367, (ref-fp64): 292582225.25993 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7515604Z E0601 05:23:02.750000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1582117620286287616.00000, (ref-fp64): 1582117620286287616.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7518129Z E0601 05:23:02.751000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 92041785.27299, (ref-fp64): 92041785.95236 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7520711Z E0601 05:23:02.751000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1621857740286198528.00000, (ref-fp64): 1621857740286198528.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7524528Z E0601 05:23:02.752000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 13134803.41969, (ref-fp64): 13134803.84750 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7528523Z E0601 05:23:02.752000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 963462929634031872.00000, (ref-fp64): 963462929634031872.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7533174Z E0601 05:23:02.752000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 58619612462221784.00000, (ref-fp64): 58619612462221784.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7537926Z E0601 05:23:02.753000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 196313275562248064.00000, (ref-fp64): 196313275562248064.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7542430Z E0601 05:23:02.753000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 67835089144001808.00000, (ref-fp64): 67835089144001808.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7546540Z E0601 05:23:02.754000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 11654732.86079, (ref-fp64): 11654733.01088 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7550849Z E0601 05:23:02.754000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1428533944575174144.00000, (ref-fp64): 1428533944575174144.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7554943Z E0601 05:23:02.755000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 3791433.67124, (ref-fp64): 3791434.38002 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7559177Z E0601 05:23:02.755000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1375826149955597056.00000, (ref-fp64): 1375826149955597056.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7563376Z E0601 05:23:02.755000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 776309.33985, (ref-fp64): 776310.00533 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7567700Z E0601 05:23:02.756000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1186998677293331456.00000, (ref-fp64): 1186998677293331456.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7572193Z E0601 05:23:02.756000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 57504690394041480.00000, (ref-fp64): 57504690394041480.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7577156Z E0601 05:23:02.757000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 199674809046237664.00000, (ref-fp64): 199674809046237664.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7581691Z E0601 05:23:02.757000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 77978445266419568.00000, (ref-fp64): 77978445266419568.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7585741Z E0601 05:23:02.758000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 262256.69502, (ref-fp64): 262257.66713 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7590084Z E0601 05:23:02.758000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 603019909294110720.00000, (ref-fp64): 603019909294110720.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7594499Z E0601 05:23:02.759000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 29506.49415, (ref-fp64): 29506.86530 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7598526Z E0601 05:23:02.759000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 501014250606529600.00000, (ref-fp64): 501014250606529600.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7602766Z E0601 05:23:02.759000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1848.00666, (ref-fp64): 1848.23230 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7607222Z E0601 05:23:02.760000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 154978656861878240.00000, (ref-fp64): 154978656861878240.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7615274Z E0601 05:23:02.761000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 24182921609577900.00000, (ref-fp64): 24182921609577900.00000 and shape=torch.Size([1024, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7619721Z E0601 05:23:02.761000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 35991615444811628.00000, (ref-fp64): 35991615444811628.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7631144Z E0601 05:23:02.762000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 9467841050185580.00000, (ref-fp64): 9467841050185580.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7642260Z E0601 05:23:02.763000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 320349294506987.25000, (ref-fp64): 320349294506987.25000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7646496Z E0601 05:23:02.764000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1848.00666, (ref-fp64): 1848.23230 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7650670Z E0601 05:23:02.764000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 7267959725199597.00000, (ref-fp64): 7267959725199597.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7654799Z E0601 05:23:02.765000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 791.26613, (ref-fp64): 791.76398 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7659076Z E0601 05:23:02.765000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 689592714139355776.00000, (ref-fp64): 689592714139355776.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7663141Z E0601 05:23:02.765000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 235.86755, (ref-fp64): 236.37316 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7667539Z E0601 05:23:02.766000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 658727893153799296.00000, (ref-fp64): 658727893153799296.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7671490Z E0601 05:23:02.766000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 13.20530, (ref-fp64): 13.45803 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7675901Z E0601 05:23:02.767000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 216863613111810656.00000, (ref-fp64): 216863613111810656.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7687674Z E0601 05:23:02.768000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 14120995187053598.00000, (ref-fp64): 14120995187053598.00000 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7691907Z E0601 05:23:02.768000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 63704817458537224.00000, (ref-fp64): 63704817458537224.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7702992Z E0601 05:23:02.769000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 14093571675526578.00000, (ref-fp64): 14093571675526578.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7707002Z E0601 05:23:02.770000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 13.67219, (ref-fp64): 14.23236 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7711332Z E0601 05:23:02.770000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 725694717256641920.00000, (ref-fp64): 725694717256641920.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7715898Z E0601 05:23:02.771000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.21527, (ref-fp64): 1.33348 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7719796Z E0601 05:23:02.771000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 562045303027117440.00000, (ref-fp64): 562045303027117440.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7724022Z E0601 05:23:02.771000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00401, (ref-fp64): 0.00495 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7728436Z E0601 05:23:02.772000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 25408374642376072.00000, (ref-fp64): 25408374642376072.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7739353Z E0601 05:23:02.773000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 15408686861042554.00000, (ref-fp64): 15408686861042554.00000 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7745243Z E0601 05:23:02.773000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 51885259565403408.00000, (ref-fp64): 51885259565403408.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7756710Z E0601 05:23:02.775000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 6382744681132833.00000, (ref-fp64): 6382744681132833.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7761552Z E0601 05:23:02.775000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01006, (ref-fp64): 0.00822 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7766286Z E0601 05:23:02.776000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01073, (ref-fp64): 0.00841 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7770667Z E0601 05:23:02.776000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01076, (ref-fp64): 0.00786 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7784067Z E0601 05:23:02.778000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00176, (ref-fp64): 0.00267 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7788751Z E0601 05:23:02.778000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00791, (ref-fp64): 0.00648 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7793072Z E0601 05:23:02.778000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00795, (ref-fp64): 0.00675 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7797507Z E0601 05:23:02.779000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01273, (ref-fp64): 0.00824 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7802052Z E0601 05:23:02.779000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01042, (ref-fp64): 0.00777 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7806695Z E0601 05:23:02.780000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01074, (ref-fp64): 0.00833 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7811209Z E0601 05:23:02.780000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01026, (ref-fp64): 0.00805 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7815771Z E0601 05:23:02.781000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00759, (ref-fp64): 0.00606 and shape=torch.Size([128, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7820247Z E0601 05:23:02.781000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00837, (ref-fp64): 0.00625 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7824781Z E0601 05:23:02.782000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00923, (ref-fp64): 0.00716 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7829307Z E0601 05:23:02.782000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00972, (ref-fp64): 0.00755 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7834051Z E0601 05:23:02.783000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01074, (ref-fp64): 0.00833 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7838317Z E0601 05:23:02.783000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01029, (ref-fp64): 0.00805 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7842922Z E0601 05:23:02.783000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01138, (ref-fp64): 0.00731 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7847504Z E0601 05:23:02.784000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01179, (ref-fp64): 0.00729 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7851798Z E0601 05:23:02.784000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01143, (ref-fp64): 0.00846 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7856122Z E0601 05:23:02.785000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01082, (ref-fp64): 0.00849 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7860698Z E0601 05:23:02.785000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01197, (ref-fp64): 0.00893 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7865116Z E0601 05:23:02.786000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01217, (ref-fp64): 0.00895 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7869632Z E0601 05:23:02.786000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00833, (ref-fp64): 0.00604 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7874358Z E0601 05:23:02.787000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00991, (ref-fp64): 0.00716 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7878985Z E0601 05:23:02.787000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00993, (ref-fp64): 0.00762 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7883586Z E0601 05:23:02.787000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01105, (ref-fp64): 0.00887 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7887959Z E0601 05:23:02.788000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01063, (ref-fp64): 0.00863 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7892457Z E0601 05:23:02.788000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01040, (ref-fp64): 0.00894 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7896722Z E0601 05:23:02.789000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01062, (ref-fp64): 0.00898 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7901097Z E0601 05:23:02.789000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01188, (ref-fp64): 0.00912 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7905530Z E0601 05:23:02.790000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01188, (ref-fp64): 0.00880 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7910468Z E0601 05:23:02.790000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00981, (ref-fp64): 0.00738 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7914999Z E0601 05:23:02.791000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01006, (ref-fp64): 0.00826 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7919461Z E0601 05:23:02.791000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01055, (ref-fp64): 0.00790 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7924002Z E0601 05:23:02.792000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01316, (ref-fp64): 0.00979 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7928580Z E0601 05:23:02.792000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01283, (ref-fp64): 0.00944 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7932788Z E0601 05:23:02.792000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01242, (ref-fp64): 0.00924 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7937123Z E0601 05:23:02.793000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01164, (ref-fp64): 0.00887 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7941828Z E0601 05:23:02.793000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01242, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7946192Z E0601 05:23:02.794000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01218, (ref-fp64): 0.00867 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7951083Z E0601 05:23:02.794000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01151, (ref-fp64): 0.00824 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7955668Z E0601 05:23:02.795000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01093, (ref-fp64): 0.00763 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.7960376Z E0601 05:23:02.795000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01011, (ref-fp64): 0.00732 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7965614Z E0601 05:23:02.796000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00996, (ref-fp64): 0.00722 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7969626Z E0601 05:23:02.796000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01242, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7974373Z E0601 05:23:02.797000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01196, (ref-fp64): 0.00865 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7978749Z E0601 05:23:02.797000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01150, (ref-fp64): 0.00920 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7983103Z E0601 05:23:02.797000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01147, (ref-fp64): 0.00840 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7987585Z E0601 05:23:02.798000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01182, (ref-fp64): 0.00956 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7992190Z E0601 05:23:02.798000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01191, (ref-fp64): 0.00947 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.7996604Z E0601 05:23:02.799000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01212, (ref-fp64): 0.00942 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8001008Z E0601 05:23:02.799000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01182, (ref-fp64): 0.00863 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8006082Z E0601 05:23:02.800000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00877, (ref-fp64): 0.00658 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8010539Z E0601 05:23:02.800000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01019, (ref-fp64): 0.00784 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8015178Z E0601 05:23:02.801000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00974, (ref-fp64): 0.00717 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8019772Z E0601 05:23:02.801000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01295, (ref-fp64): 0.00923 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8024388Z E0601 05:23:02.802000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01250, (ref-fp64): 0.00892 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8028486Z E0601 05:23:02.802000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01303, (ref-fp64): 0.00937 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8032986Z E0601 05:23:02.802000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01262, (ref-fp64): 0.00921 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8037472Z E0601 05:23:02.803000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01255, (ref-fp64): 0.00939 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8042030Z E0601 05:23:02.803000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01175, (ref-fp64): 0.00841 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8047289Z E0601 05:23:02.804000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01002, (ref-fp64): 0.00688 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8051484Z E0601 05:23:02.804000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01099, (ref-fp64): 0.00775 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8056128Z E0601 05:23:02.805000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00999, (ref-fp64): 0.00716 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8060736Z E0601 05:23:02.805000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01167, (ref-fp64): 0.00918 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8065170Z E0601 05:23:02.806000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01131, (ref-fp64): 0.00862 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8069651Z E0601 05:23:02.806000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01217, (ref-fp64): 0.00905 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8074183Z E0601 05:23:02.807000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01197, (ref-fp64): 0.00885 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8078867Z E0601 05:23:02.807000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01229, (ref-fp64): 0.00952 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8083535Z E0601 05:23:02.807000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01280, (ref-fp64): 0.00898 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8088383Z E0601 05:23:02.808000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00907, (ref-fp64): 0.00664 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8092858Z E0601 05:23:02.808000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01051, (ref-fp64): 0.00751 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8097721Z E0601 05:23:02.809000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.00707 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8102280Z E0601 05:23:02.809000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01302, (ref-fp64): 0.00903 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8106401Z E0601 05:23:02.810000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01295, (ref-fp64): 0.00913 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8110813Z E0601 05:23:02.810000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01338, (ref-fp64): 0.00943 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8115426Z E0601 05:23:02.811000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01287, (ref-fp64): 0.00842 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8119774Z E0601 05:23:02.811000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01218, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8124385Z E0601 05:23:02.812000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01175, (ref-fp64): 0.00805 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8129216Z E0601 05:23:02.812000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01009, (ref-fp64): 0.00683 and shape=torch.Size([512, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8134278Z E0601 05:23:02.813000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00961, (ref-fp64): 0.00632 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8139147Z E0601 05:23:02.813000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00943, (ref-fp64): 0.00661 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8143766Z E0601 05:23:02.813000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00902, (ref-fp64): 0.00627 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8148166Z E0601 05:23:02.814000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01218, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8152627Z E0601 05:23:02.814000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01171, (ref-fp64): 0.00799 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8157160Z E0601 05:23:02.815000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01233, (ref-fp64): 0.00860 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8161583Z E0601 05:23:02.815000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01173, (ref-fp64): 0.00805 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8166554Z E0601 05:23:02.816000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01265, (ref-fp64): 0.00923 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8170761Z E0601 05:23:02.816000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01242, (ref-fp64): 0.00850 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8175183Z E0601 05:23:02.817000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01327, (ref-fp64): 0.00906 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8179647Z E0601 05:23:02.817000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01278, (ref-fp64): 0.00870 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8184347Z E0601 05:23:02.818000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00811, (ref-fp64): 0.00609 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8189274Z E0601 05:23:02.818000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.00679 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8194325Z E0601 05:23:02.819000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00979, (ref-fp64): 0.00666 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8198824Z E0601 05:23:02.819000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01188, (ref-fp64): 0.00874 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8203177Z E0601 05:23:02.819000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01181, (ref-fp64): 0.00840 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8207832Z E0601 05:23:02.820000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01223, (ref-fp64): 0.00917 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8212125Z E0601 05:23:02.820000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01190, (ref-fp64): 0.00907 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8216568Z E0601 05:23:02.821000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01343, (ref-fp64): 0.00921 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8220900Z E0601 05:23:02.821000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01317, (ref-fp64): 0.00887 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8225796Z E0601 05:23:02.822000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00886, (ref-fp64): 0.00638 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8230914Z E0601 05:23:02.822000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00986, (ref-fp64): 0.00691 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8235867Z E0601 05:23:02.823000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00984, (ref-fp64): 0.00661 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8240265Z E0601 05:23:02.823000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01312, (ref-fp64): 0.00897 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8245039Z E0601 05:23:02.824000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01328, (ref-fp64): 0.00878 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8249378Z E0601 05:23:02.824000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01273, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8253708Z E0601 05:23:02.824000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01241, (ref-fp64): 0.00878 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8258224Z E0601 05:23:02.825000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01357, (ref-fp64): 0.00918 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8262536Z E0601 05:23:02.825000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01315, (ref-fp64): 0.00888 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8267401Z E0601 05:23:02.826000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00952, (ref-fp64): 0.00620 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8272398Z E0601 05:23:02.826000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01013, (ref-fp64): 0.00683 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8277281Z E0601 05:23:02.827000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01027, (ref-fp64): 0.00664 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8281743Z E0601 05:23:02.827000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01192, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8286464Z E0601 05:23:02.828000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01191, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8290750Z E0601 05:23:02.828000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01268, (ref-fp64): 0.00891 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8295135Z E0601 05:23:02.829000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01218, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8299551Z E0601 05:23:02.829000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01369, (ref-fp64): 0.00908 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8304021Z E0601 05:23:02.830000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01349, (ref-fp64): 0.00901 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8309002Z E0601 05:23:02.830000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00890, (ref-fp64): 0.00605 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8314025Z E0601 05:23:02.831000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00966, (ref-fp64): 0.00645 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8318566Z E0601 05:23:02.831000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01005, (ref-fp64): 0.00659 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8323187Z E0601 05:23:02.831000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01258, (ref-fp64): 0.00869 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8327569Z E0601 05:23:02.832000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01221, (ref-fp64): 0.00848 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8331916Z E0601 05:23:02.832000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01205, (ref-fp64): 0.00855 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8336528Z E0601 05:23:02.833000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01158, (ref-fp64): 0.00852 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8340808Z E0601 05:23:02.833000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01363, (ref-fp64): 0.00921 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8345143Z E0601 05:23:02.834000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01331, (ref-fp64): 0.00882 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8349897Z E0601 05:23:02.834000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00888, (ref-fp64): 0.00605 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8355088Z E0601 05:23:02.835000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00939, (ref-fp64): 0.00639 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8359591Z E0601 05:23:02.835000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00950, (ref-fp64): 0.00628 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8364173Z E0601 05:23:02.836000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01398, (ref-fp64): 0.00912 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8368647Z E0601 05:23:02.836000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01418, (ref-fp64): 0.00910 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8373004Z E0601 05:23:02.836000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01324, (ref-fp64): 0.00860 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8377451Z E0601 05:23:02.837000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01235, (ref-fp64): 0.00791 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8381834Z E0601 05:23:02.837000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01323, (ref-fp64): 0.00795 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8386272Z E0601 05:23:02.838000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01327, (ref-fp64): 0.00784 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8392954Z E0601 05:23:02.838000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00920, (ref-fp64): 0.00597 and shape=torch.Size([1024, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8397636Z E0601 05:23:02.839000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00901, (ref-fp64): 0.00556 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8409476Z E0601 05:23:02.840000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00900, (ref-fp64): 0.00566 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8420775Z E0601 05:23:02.841000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00838, (ref-fp64): 0.00543 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8425113Z E0601 05:23:02.842000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01323, (ref-fp64): 0.00795 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8429568Z E0601 05:23:02.842000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01225, (ref-fp64): 0.00784 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8434257Z E0601 05:23:02.843000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01226, (ref-fp64): 0.00797 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8438545Z E0601 05:23:02.843000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01197, (ref-fp64): 0.00794 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8442945Z E0601 05:23:02.843000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01176, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8447507Z E0601 05:23:02.844000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01142, (ref-fp64): 0.00824 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8451995Z E0601 05:23:02.844000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01198, (ref-fp64): 0.00761 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8456213Z E0601 05:23:02.845000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01166, (ref-fp64): 0.00742 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8467881Z E0601 05:23:02.846000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00803, (ref-fp64): 0.00563 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8472602Z E0601 05:23:02.846000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00788, (ref-fp64): 0.00540 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8484187Z E0601 05:23:02.848000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00760, (ref-fp64): 0.00568 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8488676Z E0601 05:23:02.848000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00990, (ref-fp64): 0.00769 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8493014Z E0601 05:23:02.848000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00981, (ref-fp64): 0.00760 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8497397Z E0601 05:23:02.849000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01370, (ref-fp64): 0.00977 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8501712Z E0601 05:23:02.849000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01179, (ref-fp64): 0.00825 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8506073Z E0601 05:23:02.850000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00962, (ref-fp64): 0.00568 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8510669Z E0601 05:23:02.850000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00777, (ref-fp64): 0.00791 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8522172Z E0601 05:23:02.851000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00575, (ref-fp64): 0.00432 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8527099Z E0601 05:23:02.852000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00666, (ref-fp64): 0.00472 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:23:02.8538598Z E0601 05:23:02.853000 140049406337664 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00704, (ref-fp64): 0.00514 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:23:02.8877443Z pass 2024-06-01T05:23:02.8992958Z TIMING: entire_frame_compile:81.05612 code_gen:22.16476 inductor_compile:45.11676 backend_compile:70.6692 2024-06-01T05:23:02.8994491Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:14353 | FakeTensorMode.__torch_dispatch__:89380 | attempt fast:1521 | fast is_contiguous:1521 | ProxyTorchDispatchMode.__torch_dispatch__:18492 2024-06-01T05:23:02.8995846Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-06-01T05:23:10.2815309Z 2024-06-01T05:23:16.2140281Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:23:16.2140755Z loading model: 0it [00:05, ?it/s] 2024-06-01T05:23:16.2141198Z cuda train sam 2024-06-01T05:23:16.2148221Z Traceback (most recent call last): 2024-06-01T05:23:16.2149126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model 2024-06-01T05:23:16.2150249Z self.model_iter_fn(model, example_inputs) 2024-06-01T05:23:16.2152262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 441, in forward_and_backward_pass 2024-06-01T05:23:16.2153325Z self.grad_scaler.scale(loss).backward() 2024-06-01T05:23:16.2154376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 520, in backward 2024-06-01T05:23:16.2155271Z torch.autograd.backward( 2024-06-01T05:23:16.2156303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 284, in backward 2024-06-01T05:23:16.2157286Z _engine_run_backward( 2024-06-01T05:23:16.2158193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward 2024-06-01T05:23:16.2159372Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-01T05:23:16.2160461Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-06-01T05:23:16.2161011Z 2024-06-01T05:23:16.2161312Z The above exception was the direct cause of the following exception: 2024-06-01T05:23:16.2161766Z 2024-06-01T05:23:16.2161920Z Traceback (most recent call last): 2024-06-01T05:23:16.2162708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run 2024-06-01T05:23:16.2163371Z ) = runner.load_model( 2024-06-01T05:23:16.2164042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model 2024-06-01T05:23:16.2164819Z self.validate_model(model, example_inputs) 2024-06-01T05:23:16.2165590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model 2024-06-01T05:23:16.2166376Z raise RuntimeError("Eager run failed") from e 2024-06-01T05:23:16.2166857Z RuntimeError: Eager run failed 2024-06-01T05:23:16.2167127Z 2024-06-01T05:23:16.2167241Z eager_fail_to_run 2024-06-01T05:23:19.4899859Z 2024-06-01T05:23:20.8257220Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:23:20.8257707Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:23:20.8258171Z cuda train shufflenet_v2_x1_0 2024-06-01T05:24:16.4539890Z E0601 05:24:16.453000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01106, (ref-fp64): 0.01401 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4546998Z E0601 05:24:16.454000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00051, (ref-fp64): 0.00048 and shape=torch.Size([24, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4551571Z E0601 05:24:16.454000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00133, (ref-fp64): 0.00112 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4556042Z E0601 05:24:16.455000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00237, (ref-fp64): 0.00257 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4560414Z E0601 05:24:16.455000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00008 and shape=torch.Size([1024, 464, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4564867Z E0601 05:24:16.456000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00044, (ref-fp64): 0.00048 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4569003Z E0601 05:24:16.456000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00055, (ref-fp64): 0.00062 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4578249Z E0601 05:24:16.457000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00049, (ref-fp64): 0.00068 and shape=torch.Size([24, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4582400Z E0601 05:24:16.457000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00165, (ref-fp64): 0.00241 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4586623Z E0601 05:24:16.458000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00162, (ref-fp64): 0.00152 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4591038Z E0601 05:24:16.458000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00049, (ref-fp64): 0.00057 and shape=torch.Size([58, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4595275Z E0601 05:24:16.459000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00128, (ref-fp64): 0.00186 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4599379Z E0601 05:24:16.459000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00112, (ref-fp64): 0.00198 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4603927Z E0601 05:24:16.459000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00060, (ref-fp64): 0.00057 and shape=torch.Size([58, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4608215Z E0601 05:24:16.460000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00059, (ref-fp64): 0.00055 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4612129Z E0601 05:24:16.460000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00180, (ref-fp64): 0.00106 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4616441Z E0601 05:24:16.461000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00020, (ref-fp64): 0.00018 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4620437Z E0601 05:24:16.461000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00054 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4624609Z E0601 05:24:16.462000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00069 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4629133Z E0601 05:24:16.462000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00034, (ref-fp64): 0.00026 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4633242Z E0601 05:24:16.462000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00166, (ref-fp64): 0.00110 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4637518Z E0601 05:24:16.463000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00137, (ref-fp64): 0.00096 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4641762Z E0601 05:24:16.463000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00031, (ref-fp64): 0.00039 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4646185Z E0601 05:24:16.464000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00145 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4650326Z E0601 05:24:16.464000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00133, (ref-fp64): 0.00097 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4654446Z E0601 05:24:16.465000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00029, (ref-fp64): 0.00022 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4658529Z E0601 05:24:16.465000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00083, (ref-fp64): 0.00085 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4662580Z E0601 05:24:16.465000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00094, (ref-fp64): 0.00063 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4666962Z E0601 05:24:16.466000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00027, (ref-fp64): 0.00029 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4671580Z E0601 05:24:16.466000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00112, (ref-fp64): 0.00121 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4675625Z E0601 05:24:16.467000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00125, (ref-fp64): 0.00112 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4679906Z E0601 05:24:16.467000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00055, (ref-fp64): 0.00033 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4683973Z E0601 05:24:16.467000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00173, (ref-fp64): 0.00102 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4688315Z E0601 05:24:16.468000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00119, (ref-fp64): 0.00085 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4694481Z E0601 05:24:16.469000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00162, (ref-fp64): 0.00117 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4698661Z E0601 05:24:16.469000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00066, (ref-fp64): 0.00053 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4704793Z E0601 05:24:16.470000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00176, (ref-fp64): 0.00144 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4708924Z E0601 05:24:16.470000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00074, (ref-fp64): 0.00056 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4713698Z E0601 05:24:16.470000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00037, (ref-fp64): 0.00045 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4719476Z E0601 05:24:16.471000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00228, (ref-fp64): 0.00243 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4722312Z E0601 05:24:16.471000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00163, (ref-fp64): 0.00184 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4726320Z E0601 05:24:16.472000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00036, (ref-fp64): 0.00032 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4730406Z E0601 05:24:16.472000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00121 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4734553Z E0601 05:24:16.473000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00083, (ref-fp64): 0.00093 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4738866Z E0601 05:24:16.473000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00026, (ref-fp64): 0.00027 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4743016Z E0601 05:24:16.473000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00153, (ref-fp64): 0.00192 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4747212Z E0601 05:24:16.474000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00137, (ref-fp64): 0.00169 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4753702Z E0601 05:24:16.474000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00076, (ref-fp64): 0.00080 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4757684Z E0601 05:24:16.475000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00070, (ref-fp64): 0.00077 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4762455Z E0601 05:24:16.475000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00026, (ref-fp64): 0.00027 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4766697Z E0601 05:24:16.476000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00093, (ref-fp64): 0.00111 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4770621Z E0601 05:24:16.476000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00092, (ref-fp64): 0.00095 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4775138Z E0601 05:24:16.477000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00043, (ref-fp64): 0.00044 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4779163Z E0601 05:24:16.477000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00140, (ref-fp64): 0.00147 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4783167Z E0601 05:24:16.477000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00173, (ref-fp64): 0.00174 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4789908Z E0601 05:24:16.478000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00067, (ref-fp64): 0.00073 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4794115Z E0601 05:24:16.478000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00083, (ref-fp64): 0.00073 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4798249Z E0601 05:24:16.479000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00022, (ref-fp64): 0.00024 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4802415Z E0601 05:24:16.479000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00087, (ref-fp64): 0.00107 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4806800Z E0601 05:24:16.480000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00083, (ref-fp64): 0.00088 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4811051Z E0601 05:24:16.480000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00021, (ref-fp64): 0.00022 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4815307Z E0601 05:24:16.481000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00061, (ref-fp64): 0.00064 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4819489Z E0601 05:24:16.481000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00109, (ref-fp64): 0.00093 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4823776Z E0601 05:24:16.481000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00040, (ref-fp64): 0.00039 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4827898Z E0601 05:24:16.482000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00098 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4832147Z E0601 05:24:16.482000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00069, (ref-fp64): 0.00054 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4836604Z E0601 05:24:16.483000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00025, (ref-fp64): 0.00025 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4840680Z E0601 05:24:16.483000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00137, (ref-fp64): 0.00140 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4845022Z E0601 05:24:16.484000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00066, (ref-fp64): 0.00069 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4849487Z E0601 05:24:16.484000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00033, (ref-fp64): 0.00037 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4853436Z E0601 05:24:16.484000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00172, (ref-fp64): 0.00195 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4857568Z E0601 05:24:16.485000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00172, (ref-fp64): 0.00197 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4861882Z E0601 05:24:16.485000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00024, (ref-fp64): 0.00036 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4866135Z E0601 05:24:16.486000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00095, (ref-fp64): 0.00105 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4870287Z E0601 05:24:16.486000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00104 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4874665Z E0601 05:24:16.487000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00025, (ref-fp64): 0.00026 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4878776Z E0601 05:24:16.487000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00136 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4882871Z E0601 05:24:16.487000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00119, (ref-fp64): 0.00159 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4887471Z E0601 05:24:16.488000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00038, (ref-fp64): 0.00041 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4891500Z E0601 05:24:16.488000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00151, (ref-fp64): 0.00162 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4895682Z E0601 05:24:16.489000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00143, (ref-fp64): 0.00157 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4901870Z E0601 05:24:16.489000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00100, (ref-fp64): 0.00105 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4905884Z E0601 05:24:16.490000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00075, (ref-fp64): 0.00075 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4910582Z E0601 05:24:16.490000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00022, (ref-fp64): 0.00024 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4914681Z E0601 05:24:16.491000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00133, (ref-fp64): 0.00151 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4918780Z E0601 05:24:16.491000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00084, (ref-fp64): 0.00096 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4923264Z E0601 05:24:16.491000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00034, (ref-fp64): 0.00037 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4927363Z E0601 05:24:16.492000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00166, (ref-fp64): 0.00178 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4931453Z E0601 05:24:16.492000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00142 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4937757Z E0601 05:24:16.493000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00121, (ref-fp64): 0.00152 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4941882Z E0601 05:24:16.493000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00076, (ref-fp64): 0.00086 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4946088Z E0601 05:24:16.494000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00023, (ref-fp64): 0.00024 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4950397Z E0601 05:24:16.494000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00143, (ref-fp64): 0.00176 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4954589Z E0601 05:24:16.495000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00095, (ref-fp64): 0.00103 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4958988Z E0601 05:24:16.495000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00036, (ref-fp64): 0.00038 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4963340Z E0601 05:24:16.495000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00165, (ref-fp64): 0.00166 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4967796Z E0601 05:24:16.496000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00095, (ref-fp64): 0.00098 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4972151Z E0601 05:24:16.496000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00019, (ref-fp64): 0.00019 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:16.4976256Z E0601 05:24:16.497000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00118, (ref-fp64): 0.00132 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4980422Z E0601 05:24:16.497000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00061, (ref-fp64): 0.00058 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4984815Z E0601 05:24:16.498000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00020, (ref-fp64): 0.00021 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4988925Z E0601 05:24:16.498000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00192, (ref-fp64): 0.00184 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4993098Z E0601 05:24:16.498000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00129, (ref-fp64): 0.00122 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.4997713Z E0601 05:24:16.499000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00035, (ref-fp64): 0.00042 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5001834Z E0601 05:24:16.499000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00129, (ref-fp64): 0.00152 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5006342Z E0601 05:24:16.500000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00119 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5012421Z E0601 05:24:16.500000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00084, (ref-fp64): 0.00106 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5016627Z E0601 05:24:16.501000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00067, (ref-fp64): 0.00082 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5020873Z E0601 05:24:16.501000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00023, (ref-fp64): 0.00027 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5024993Z E0601 05:24:16.502000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00111, (ref-fp64): 0.00137 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5029058Z E0601 05:24:16.502000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00100, (ref-fp64): 0.00133 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5033612Z E0601 05:24:16.502000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00033, (ref-fp64): 0.00047 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5037688Z E0601 05:24:16.503000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00121, (ref-fp64): 0.00163 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5041797Z E0601 05:24:16.503000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00094, (ref-fp64): 0.00136 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5048423Z E0601 05:24:16.504000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00071, (ref-fp64): 0.00090 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5052422Z E0601 05:24:16.504000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00065, (ref-fp64): 0.00076 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5056741Z E0601 05:24:16.505000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00017, (ref-fp64): 0.00020 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5060753Z E0601 05:24:16.505000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00119, (ref-fp64): 0.00165 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5064860Z E0601 05:24:16.506000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00112 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5071408Z E0601 05:24:16.506000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00078, (ref-fp64): 0.00080 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5075560Z E0601 05:24:16.507000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00095, (ref-fp64): 0.00091 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5080125Z E0601 05:24:16.507000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00024, (ref-fp64): 0.00024 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5084616Z E0601 05:24:16.508000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00102, (ref-fp64): 0.00102 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5088789Z E0601 05:24:16.508000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00115, (ref-fp64): 0.00109 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5093364Z E0601 05:24:16.508000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00013, (ref-fp64): 0.00030 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5097641Z E0601 05:24:16.509000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00048, (ref-fp64): 0.00110 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5101646Z E0601 05:24:16.509000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00056, (ref-fp64): 0.00168 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5107911Z E0601 05:24:16.510000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00050, (ref-fp64): 0.00066 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5112096Z E0601 05:24:16.510000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00035, (ref-fp64): 0.00058 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5116935Z E0601 05:24:16.511000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00011, (ref-fp64): 0.00014 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5120968Z E0601 05:24:16.511000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00069, (ref-fp64): 0.00088 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5125433Z E0601 05:24:16.512000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00047, (ref-fp64): 0.00064 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5129929Z E0601 05:24:16.512000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00023, (ref-fp64): 0.00028 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5134474Z E0601 05:24:16.513000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00107, (ref-fp64): 0.00123 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5138672Z E0601 05:24:16.513000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00111, (ref-fp64): 0.00120 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5144986Z E0601 05:24:16.514000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00068, (ref-fp64): 0.00074 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5149166Z E0601 05:24:16.514000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00069, (ref-fp64): 0.00067 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5153977Z E0601 05:24:16.514000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00017, (ref-fp64): 0.00016 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5158195Z E0601 05:24:16.515000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00086, (ref-fp64): 0.00086 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5162473Z E0601 05:24:16.515000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00060, (ref-fp64): 0.00059 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5167091Z E0601 05:24:16.516000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00016, (ref-fp64): 0.00015 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5171265Z E0601 05:24:16.516000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00085, (ref-fp64): 0.00083 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5175554Z E0601 05:24:16.517000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00084, (ref-fp64): 0.00071 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5181691Z E0601 05:24:16.517000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00047, (ref-fp64): 0.00044 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5185817Z E0601 05:24:16.518000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00043, (ref-fp64): 0.00037 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5190538Z E0601 05:24:16.518000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00011, (ref-fp64): 0.00011 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5194667Z E0601 05:24:16.519000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00056, (ref-fp64): 0.00050 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5198746Z E0601 05:24:16.519000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00042, (ref-fp64): 0.00045 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5203451Z E0601 05:24:16.519000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00015, (ref-fp64): 0.00016 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5207541Z E0601 05:24:16.520000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00075, (ref-fp64): 0.00085 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5211728Z E0601 05:24:16.520000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00087, (ref-fp64): 0.00092 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5218306Z E0601 05:24:16.521000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00038, (ref-fp64): 0.00041 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5222554Z E0601 05:24:16.521000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00044, (ref-fp64): 0.00039 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5226968Z E0601 05:24:16.522000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00010, (ref-fp64): 0.00011 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5231159Z E0601 05:24:16.522000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00040, (ref-fp64): 0.00045 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5235390Z E0601 05:24:16.523000 140094927123072 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00030, (ref-fp64): 0.00034 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:24:16.5939371Z pass 2024-06-01T05:24:16.5951003Z TIMING: entire_frame_compile:51.54627 code_gen:14.66009 inductor_compile:31.46331 backend_compile:47.57179 2024-06-01T05:24:16.5952566Z STATS: call_* op count: 307 | FakeTensor.__torch_dispatch__:7857 | FakeTensorMode.__torch_dispatch__:53616 | attempt fast:1628 | fast is_contiguous:1628 | ProxyTorchDispatchMode.__torch_dispatch__:12199 2024-06-01T05:24:16.5953954Z Dynamo produced 2 graphs covering 307 ops with 6 graph breaks (5 unique) 2024-06-01T05:24:22.4760596Z 2024-06-01T05:24:22.9813512Z loading model: 0it [00:00, ?it/s]/opt/conda/envs/py_3.10/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`. (Deprecated NumPy 1.24) 2024-06-01T05:24:22.9815593Z if not isinstance(terminated, (bool, np.bool8)): 2024-06-01T05:24:23.2759175Z 2024-06-01T05:24:23.2760099Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:24:23.2760922Z cuda train soft_actor_critic 2024-06-01T05:24:30.4674137Z W0601 05:24:30.466000 140151263433344 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:24:33.6159353Z E0601 05:24:33.615000 140151263433344 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01210, (ref-fp64): 0.01151 and shape=torch.Size([2, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:24:33.6177187Z pass 2024-06-01T05:24:33.6186270Z TIMING: entire_frame_compile:7.9823 code_gen:3.19288 inductor_compile:4.19331 backend_compile:7.20344 2024-06-01T05:24:33.6187904Z STATS: call_* op count: 59 | FakeTensor.__torch_dispatch__:574 | FakeTensorMode.__torch_dispatch__:4511 | attempt fast:70 | fast is_contiguous:70 | ProxyTorchDispatchMode.__torch_dispatch__:894 2024-06-01T05:24:33.6189352Z Dynamo produced 4 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-06-01T05:24:37.3261588Z 2024-06-01T05:24:38.5547302Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:24:38.5548494Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:24:38.5549635Z Traceback (most recent call last): 2024-06-01T05:24:38.5551995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in 2024-06-01T05:24:38.5552933Z torchbench_main() 2024-06-01T05:24:38.5553635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 452, in torchbench_main 2024-06-01T05:24:38.5554539Z main(TorchBenchmarkRunner(), original_dir) 2024-06-01T05:24:38.5555491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3660, in main 2024-06-01T05:24:38.5556482Z process_entry(0, runner, original_dir, args) 2024-06-01T05:24:38.5557591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3592, in process_entry 2024-06-01T05:24:38.5558389Z return run(runner, args, original_dir) 2024-06-01T05:24:38.5559095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4193, in run 2024-06-01T05:24:38.5562703Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-01T05:24:38.5563510Z AssertionError: nothing in example_inputs had a dim with 32 2024-06-01T05:24:39.1890950Z Run failed with return code: 1 2024-06-01T05:24:39.1891525Z Output: None 2024-06-01T05:24:39.1891824Z Error: None 2024-06-01T05:24:41.7823554Z 2024-06-01T05:24:43.0904624Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:24:43.0905113Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:24:43.0905567Z cuda train squeezenet1_1 2024-06-01T05:25:06.0254829Z E0601 05:25:06.024000 140374786134656 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00125, (ref-fp64): 0.00123 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:25:06.0454065Z pass 2024-06-01T05:25:06.0476141Z TIMING: entire_frame_compile:19.86297 code_gen:10.43927 inductor_compile:14.54395 backend_compile:18.81038 2024-06-01T05:25:06.0479458Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:1285 | FakeTensorMode.__torch_dispatch__:11556 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | attempt fast:16 | fast is_contiguous:16 2024-06-01T05:25:06.0480959Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-06-01T05:25:10.2735408Z 2024-06-01T05:25:11.3489183Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:25:11.3489616Z 2024-06-01T05:25:11.6345679Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-06-01T05:28:43.0072260Z E0601 05:28:43.006000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00567, (ref-fp64): 0.00569 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0516430Z E0601 05:28:43.050000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01397, (ref-fp64): 0.01856 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0521916Z E0601 05:28:43.051000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01057, (ref-fp64): 0.01602 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0524290Z E0601 05:28:43.051000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01645, (ref-fp64): 0.02334 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0526621Z E0601 05:28:43.052000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00958, (ref-fp64): 0.01568 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0531653Z E0601 05:28:43.052000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00983, (ref-fp64): 0.01409 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0535851Z E0601 05:28:43.053000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01009, (ref-fp64): 0.01638 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0540365Z E0601 05:28:43.053000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00908, (ref-fp64): 0.01267 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0545189Z E0601 05:28:43.054000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00479, (ref-fp64): 0.00490 and shape=torch.Size([32, 8, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0549599Z E0601 05:28:43.054000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00937, (ref-fp64): 0.01266 and shape=torch.Size([8]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0556443Z E0601 05:28:43.055000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01363, (ref-fp64): 0.01863 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0560796Z E0601 05:28:43.055000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00887, (ref-fp64): 0.00835 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0565478Z E0601 05:28:43.056000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02026, (ref-fp64): 0.01682 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0569732Z E0601 05:28:43.056000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01066, (ref-fp64): 0.00829 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0574363Z E0601 05:28:43.057000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02208, (ref-fp64): 0.01119 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0578780Z E0601 05:28:43.057000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01288, (ref-fp64): 0.00981 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0583396Z E0601 05:28:43.057000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00813, (ref-fp64): 0.01198 and shape=torch.Size([96, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0588030Z E0601 05:28:43.058000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00929, (ref-fp64): 0.01541 and shape=torch.Size([96, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0592980Z E0601 05:28:43.058000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00806, (ref-fp64): 0.01143 and shape=torch.Size([24, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0605563Z E0601 05:28:43.060000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01369, (ref-fp64): 0.01687 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0610083Z E0601 05:28:43.060000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00724, (ref-fp64): 0.00699 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0614587Z E0601 05:28:43.061000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02017, (ref-fp64): 0.01532 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0619136Z E0601 05:28:43.061000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00800, (ref-fp64): 0.00709 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0623528Z E0601 05:28:43.061000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02022, (ref-fp64): 0.01604 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0628187Z E0601 05:28:43.062000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01193, (ref-fp64): 0.00782 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0632885Z E0601 05:28:43.062000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00720, (ref-fp64): 0.00821 and shape=torch.Size([144, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0638603Z E0601 05:28:43.063000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00758, (ref-fp64): 0.01118 and shape=torch.Size([144, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0642679Z E0601 05:28:43.063000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00709, (ref-fp64): 0.00721 and shape=torch.Size([24, 144, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0655563Z E0601 05:28:43.065000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01634, (ref-fp64): 0.01604 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0659868Z E0601 05:28:43.065000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00991, (ref-fp64): 0.00892 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0664385Z E0601 05:28:43.066000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01267, (ref-fp64): 0.01248 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0668954Z E0601 05:28:43.066000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01054, (ref-fp64): 0.00884 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0673819Z E0601 05:28:43.066000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01252, (ref-fp64): 0.01268 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0678205Z E0601 05:28:43.067000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01260, (ref-fp64): 0.01014 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0683019Z E0601 05:28:43.067000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00836, (ref-fp64): 0.00936 and shape=torch.Size([144, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0688031Z E0601 05:28:43.068000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01124, (ref-fp64): 0.01266 and shape=torch.Size([144, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0692718Z E0601 05:28:43.068000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00833, (ref-fp64): 0.00832 and shape=torch.Size([40, 144, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0697141Z E0601 05:28:43.069000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00821, (ref-fp64): 0.00809 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0707744Z E0601 05:28:43.070000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01543, (ref-fp64): 0.01569 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0712565Z E0601 05:28:43.070000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00802, (ref-fp64): 0.00703 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0717007Z E0601 05:28:43.071000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01066, (ref-fp64): 0.01343 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0721538Z E0601 05:28:43.071000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00795, (ref-fp64): 0.00698 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0726521Z E0601 05:28:43.072000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01267, (ref-fp64): 0.01348 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0730668Z E0601 05:28:43.072000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01007, (ref-fp64): 0.00829 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0735416Z E0601 05:28:43.073000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00706, (ref-fp64): 0.00691 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0740166Z E0601 05:28:43.073000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00761, (ref-fp64): 0.00773 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0744807Z E0601 05:28:43.074000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00723, (ref-fp64): 0.00744 and shape=torch.Size([40, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0749280Z E0601 05:28:43.074000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00704, (ref-fp64): 0.00709 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0760256Z E0601 05:28:43.075000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01329, (ref-fp64): 0.01466 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0764803Z E0601 05:28:43.076000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00942, (ref-fp64): 0.00805 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0769449Z E0601 05:28:43.076000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00988, (ref-fp64): 0.01012 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0774016Z E0601 05:28:43.077000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00832, (ref-fp64): 0.00810 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0779036Z E0601 05:28:43.077000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00968, (ref-fp64): 0.01113 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0783252Z E0601 05:28:43.077000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00851, (ref-fp64): 0.00949 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0787777Z E0601 05:28:43.078000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00823, (ref-fp64): 0.00823 and shape=torch.Size([240, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0792334Z E0601 05:28:43.078000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00957, (ref-fp64): 0.00945 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0797230Z E0601 05:28:43.079000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00804, (ref-fp64): 0.00811 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0801744Z E0601 05:28:43.079000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00781 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0806703Z E0601 05:28:43.080000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00582, (ref-fp64): 0.00584 and shape=torch.Size([240, 10, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0811026Z E0601 05:28:43.080000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00681, (ref-fp64): 0.00787 and shape=torch.Size([10]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0817649Z E0601 05:28:43.081000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01125, (ref-fp64): 0.01274 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0822284Z E0601 05:28:43.081000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0826815Z E0601 05:28:43.082000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01016, (ref-fp64): 0.01063 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0831458Z E0601 05:28:43.082000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00752, (ref-fp64): 0.00713 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0836037Z E0601 05:28:43.083000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00961, (ref-fp64): 0.01104 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0840421Z E0601 05:28:43.083000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00821, (ref-fp64): 0.00781 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0845508Z E0601 05:28:43.084000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00749, (ref-fp64): 0.00766 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0850017Z E0601 05:28:43.084000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00754, (ref-fp64): 0.00773 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0862547Z E0601 05:28:43.085000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00720, (ref-fp64): 0.00754 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0864874Z E0601 05:28:43.085000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00713, (ref-fp64): 0.00721 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0867456Z E0601 05:28:43.086000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00496, (ref-fp64): 0.00496 and shape=torch.Size([480, 20, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0869588Z E0601 05:28:43.086000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00663, (ref-fp64): 0.00733 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0875522Z E0601 05:28:43.087000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01087, (ref-fp64): 0.01278 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0879846Z E0601 05:28:43.087000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00773, (ref-fp64): 0.00720 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0884633Z E0601 05:28:43.088000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00975, (ref-fp64): 0.01012 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0889080Z E0601 05:28:43.088000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00742, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0893561Z E0601 05:28:43.088000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01044, (ref-fp64): 0.01065 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0898099Z E0601 05:28:43.089000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00906, (ref-fp64): 0.00803 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0902797Z E0601 05:28:43.089000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00730, (ref-fp64): 0.00757 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0907517Z E0601 05:28:43.090000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00778, (ref-fp64): 0.00811 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0912328Z E0601 05:28:43.090000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00771, (ref-fp64): 0.00790 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0916883Z E0601 05:28:43.091000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00707, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0921526Z E0601 05:28:43.091000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00555, (ref-fp64): 0.00556 and shape=torch.Size([480, 20, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0926209Z E0601 05:28:43.092000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00670, (ref-fp64): 0.00736 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0932815Z E0601 05:28:43.092000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01057, (ref-fp64): 0.01108 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0937235Z E0601 05:28:43.093000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00876, (ref-fp64): 0.00790 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0941830Z E0601 05:28:43.093000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01064, (ref-fp64): 0.01091 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0946301Z E0601 05:28:43.094000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00864, (ref-fp64): 0.00791 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0951130Z E0601 05:28:43.094000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01061, (ref-fp64): 0.01044 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0955615Z E0601 05:28:43.095000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00903, (ref-fp64): 0.00932 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0960381Z E0601 05:28:43.095000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00879, (ref-fp64): 0.00906 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.0965514Z E0601 05:28:43.096000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00915, (ref-fp64): 0.00929 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0970175Z E0601 05:28:43.096000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00851, (ref-fp64): 0.00874 and shape=torch.Size([112, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0974583Z E0601 05:28:43.097000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00820, (ref-fp64): 0.00763 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0979415Z E0601 05:28:43.097000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00664, (ref-fp64): 0.00657 and shape=torch.Size([480, 20, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0983912Z E0601 05:28:43.098000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00759, (ref-fp64): 0.00928 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0988678Z E0601 05:28:43.098000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00605, (ref-fp64): 0.00607 and shape=torch.Size([20, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0993267Z E0601 05:28:43.098000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01082, (ref-fp64): 0.01079 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.0997757Z E0601 05:28:43.099000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00852, (ref-fp64): 0.00736 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1002391Z E0601 05:28:43.099000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01060, (ref-fp64): 0.01073 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1007100Z E0601 05:28:43.100000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00852, (ref-fp64): 0.00735 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1011560Z E0601 05:28:43.100000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01034, (ref-fp64): 0.01064 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1016180Z E0601 05:28:43.101000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00888, (ref-fp64): 0.00771 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1020959Z E0601 05:28:43.101000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00842, (ref-fp64): 0.00857 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1026061Z E0601 05:28:43.102000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00859, (ref-fp64): 0.00861 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1031601Z E0601 05:28:43.102000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00818, (ref-fp64): 0.00831 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1036016Z E0601 05:28:43.103000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00761, (ref-fp64): 0.00733 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1040728Z E0601 05:28:43.103000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00582, (ref-fp64): 0.00583 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1045431Z E0601 05:28:43.104000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00781, (ref-fp64): 0.00837 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1050103Z E0601 05:28:43.104000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00519, (ref-fp64): 0.00519 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1054691Z E0601 05:28:43.105000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01040, (ref-fp64): 0.01137 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1059061Z E0601 05:28:43.105000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00837, (ref-fp64): 0.00746 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1063738Z E0601 05:28:43.105000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01016, (ref-fp64): 0.01046 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1068243Z E0601 05:28:43.106000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00832, (ref-fp64): 0.00746 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1072891Z E0601 05:28:43.106000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01044, (ref-fp64): 0.01080 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1077398Z E0601 05:28:43.107000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00898, (ref-fp64): 0.00781 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1082015Z E0601 05:28:43.107000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00850, (ref-fp64): 0.00877 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1087390Z E0601 05:28:43.108000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00864, (ref-fp64): 0.00879 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1092506Z E0601 05:28:43.108000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00829, (ref-fp64): 0.00844 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1096868Z E0601 05:28:43.109000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00734 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1101570Z E0601 05:28:43.109000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00621, (ref-fp64): 0.00627 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1106032Z E0601 05:28:43.110000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00794, (ref-fp64): 0.00908 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1110975Z E0601 05:28:43.110000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00584, (ref-fp64): 0.00586 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1115496Z E0601 05:28:43.111000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01112, (ref-fp64): 0.01100 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1119993Z E0601 05:28:43.111000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.00813 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1124776Z E0601 05:28:43.112000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01088, (ref-fp64): 0.01086 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1129341Z E0601 05:28:43.112000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00958, (ref-fp64): 0.00832 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1133767Z E0601 05:28:43.112000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01055, (ref-fp64): 0.01081 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1138234Z E0601 05:28:43.113000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00955, (ref-fp64): 0.01019 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1143056Z E0601 05:28:43.113000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00966, (ref-fp64): 0.00975 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1148152Z E0601 05:28:43.114000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01015, (ref-fp64): 0.01019 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1153797Z E0601 05:28:43.114000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00933, (ref-fp64): 0.00946 and shape=torch.Size([192, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1158171Z E0601 05:28:43.115000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00951, (ref-fp64): 0.00869 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1162917Z E0601 05:28:43.115000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00765, (ref-fp64): 0.00742 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1167594Z E0601 05:28:43.116000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00798, (ref-fp64): 0.00830 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1172303Z E0601 05:28:43.116000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00645, (ref-fp64): 0.00646 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1176749Z E0601 05:28:43.117000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01122, (ref-fp64): 0.01133 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1181571Z E0601 05:28:43.117000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00941, (ref-fp64): 0.00743 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1185913Z E0601 05:28:43.118000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01115, (ref-fp64): 0.01113 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1190600Z E0601 05:28:43.118000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00947, (ref-fp64): 0.00749 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1195265Z E0601 05:28:43.119000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01111, (ref-fp64): 0.01129 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1199659Z E0601 05:28:43.119000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01006, (ref-fp64): 0.00879 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1204601Z E0601 05:28:43.120000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.00962 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1209495Z E0601 05:28:43.120000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00961, (ref-fp64): 0.00969 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1214221Z E0601 05:28:43.121000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00934, (ref-fp64): 0.00938 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1218682Z E0601 05:28:43.121000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00895, (ref-fp64): 0.00781 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1223581Z E0601 05:28:43.121000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00651, (ref-fp64): 0.00644 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1227968Z E0601 05:28:43.122000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00819, (ref-fp64): 0.00859 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1233066Z E0601 05:28:43.122000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00548, (ref-fp64): 0.00549 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1237514Z E0601 05:28:43.123000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01114, (ref-fp64): 0.01133 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1242325Z E0601 05:28:43.123000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.00761 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1247049Z E0601 05:28:43.124000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01110, (ref-fp64): 0.01114 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1251692Z E0601 05:28:43.124000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00958, (ref-fp64): 0.00764 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1256294Z E0601 05:28:43.125000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01094, (ref-fp64): 0.01100 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1260832Z E0601 05:28:43.125000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01004, (ref-fp64): 0.00902 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1265507Z E0601 05:28:43.126000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00965, (ref-fp64): 0.00969 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1270530Z E0601 05:28:43.126000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00971, (ref-fp64): 0.00977 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1275585Z E0601 05:28:43.127000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00940, (ref-fp64): 0.00944 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1280039Z E0601 05:28:43.127000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00915, (ref-fp64): 0.00810 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1285132Z E0601 05:28:43.128000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00670, (ref-fp64): 0.00654 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1289609Z E0601 05:28:43.128000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.00964 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1294452Z E0601 05:28:43.129000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00571, (ref-fp64): 0.00567 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1299011Z E0601 05:28:43.129000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01101, (ref-fp64): 0.01108 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1303491Z E0601 05:28:43.129000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.00773 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1307988Z E0601 05:28:43.130000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01077, (ref-fp64): 0.01073 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1312652Z E0601 05:28:43.130000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00954, (ref-fp64): 0.00770 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1317175Z E0601 05:28:43.131000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01088, (ref-fp64): 0.01084 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1321573Z E0601 05:28:43.131000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01017, (ref-fp64): 0.00903 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1326661Z E0601 05:28:43.132000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00934, (ref-fp64): 0.00939 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1331431Z E0601 05:28:43.132000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00969, (ref-fp64): 0.00972 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1336322Z E0601 05:28:43.133000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00949, (ref-fp64): 0.00954 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1340717Z E0601 05:28:43.133000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00899, (ref-fp64): 0.00826 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1345669Z E0601 05:28:43.134000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00661, (ref-fp64): 0.00648 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1350123Z E0601 05:28:43.134000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00947, (ref-fp64): 0.00944 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1355101Z E0601 05:28:43.135000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00613, (ref-fp64): 0.00609 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1359601Z E0601 05:28:43.135000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01100, (ref-fp64): 0.01103 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1364368Z E0601 05:28:43.136000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00991, (ref-fp64): 0.00897 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1368971Z E0601 05:28:43.136000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01104, (ref-fp64): 0.01105 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1373530Z E0601 05:28:43.136000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00992, (ref-fp64): 0.00910 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1378067Z E0601 05:28:43.137000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01085, (ref-fp64): 0.01088 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1382633Z E0601 05:28:43.137000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01020, (ref-fp64): 0.01078 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1387513Z E0601 05:28:43.138000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01015, (ref-fp64): 0.01013 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1392341Z E0601 05:28:43.138000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01005, (ref-fp64): 0.01009 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1397241Z E0601 05:28:43.139000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01014, (ref-fp64): 0.01016 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1401797Z E0601 05:28:43.139000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00984, (ref-fp64): 0.00937 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1407097Z E0601 05:28:43.140000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00870, (ref-fp64): 0.00849 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1411450Z E0601 05:28:43.140000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01006, (ref-fp64): 0.01003 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1416306Z E0601 05:28:43.141000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00804, (ref-fp64): 0.00806 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1420788Z E0601 05:28:43.141000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01269, (ref-fp64): 0.01892 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1425189Z E0601 05:28:43.142000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01075, (ref-fp64): 0.01528 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1429772Z E0601 05:28:43.142000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01013, (ref-fp64): 0.01011 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1434513Z E0601 05:28:43.143000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00997, (ref-fp64): 0.00957 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1444687Z E0601 05:28:43.144000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00331, (ref-fp64): 0.00361 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:28:43.1449496Z E0601 05:28:43.144000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00996, (ref-fp64): 0.01000 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1454247Z E0601 05:28:43.145000 140247439200896 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01008, (ref-fp64): 0.01739 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:28:43.1776128Z pass 2024-06-01T05:28:43.1827865Z TIMING: entire_frame_compile:126.3223 code_gen:32.44741 inductor_compile:69.42522 backend_compile:107.37257 2024-06-01T05:28:43.1830548Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:20092 | FakeTensorMode.__torch_dispatch__:123546 | attempt fast:2535 | fast is_contiguous:2535 | ProxyTorchDispatchMode.__torch_dispatch__:24017 2024-06-01T05:28:43.1832093Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-06-01T05:28:52.9648113Z 2024-06-01T05:28:55.8553330Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:28:55.8553964Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:28:55.8555861Z cuda train timm_regnet 2024-06-01T05:30:24.3218843Z pass 2024-06-01T05:30:24.3594937Z TIMING: entire_frame_compile:72.28269 code_gen:13.60742 inductor_compile:37.88017 backend_compile:62.53142 2024-06-01T05:30:24.3596511Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:9972 | FakeTensorMode.__torch_dispatch__:74755 | attempt fast:2248 | fast is_contiguous:2248 | ProxyTorchDispatchMode.__torch_dispatch__:17103 2024-06-01T05:30:24.3597868Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-06-01T05:30:31.5971249Z 2024-06-01T05:30:33.6919534Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:30:33.6920130Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:30:33.6920637Z cuda train timm_resnest 2024-06-01T05:31:16.9453512Z W0601 05:31:16.944000 140407079572096 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:31:41.2962075Z E0601 05:31:41.295000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4.85162, (ref-fp64): 5.08572 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.2968228Z E0601 05:31:41.296000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 4.84127, (ref-fp64): 5.07537 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2973534Z E0601 05:31:41.296000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.38731, (ref-fp64): 0.31240 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2978355Z E0601 05:31:41.297000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.43208, (ref-fp64): 0.34860 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2983509Z E0601 05:31:41.297000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.17536, (ref-fp64): 0.14320 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2988782Z E0601 05:31:41.298000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.91083, (ref-fp64): 0.73894 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2993277Z E0601 05:31:41.298000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.37014, (ref-fp64): 0.29438 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.2998848Z E0601 05:31:41.299000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.16878, (ref-fp64): 0.12461 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3004062Z E0601 05:31:41.299000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.93885, (ref-fp64): 0.68664 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3008970Z E0601 05:31:41.300000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.91478, (ref-fp64): 0.69727 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3013905Z E0601 05:31:41.300000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.21487, (ref-fp64): 0.17460 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3026493Z E0601 05:31:41.302000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00112, (ref-fp64): 0.00209 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3031527Z E0601 05:31:41.302000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00130, (ref-fp64): 0.00309 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3036650Z E0601 05:31:41.303000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.10787, (ref-fp64): 0.08913 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3041673Z E0601 05:31:41.303000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02950, (ref-fp64): 0.02501 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3046992Z E0601 05:31:41.304000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00108, (ref-fp64): 0.00172 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3052167Z E0601 05:31:41.304000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00137, (ref-fp64): 0.00266 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3057245Z E0601 05:31:41.305000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00087, (ref-fp64): 0.00162 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3062027Z E0601 05:31:41.305000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00065, (ref-fp64): 0.00075 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3066835Z E0601 05:31:41.306000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00029, (ref-fp64): 0.00042 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3072143Z E0601 05:31:41.306000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00122, (ref-fp64): 0.00204 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3077083Z E0601 05:31:41.307000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00065, (ref-fp64): 0.00075 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3082336Z E0601 05:31:41.307000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00045, (ref-fp64): 0.00052 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3087586Z E0601 05:31:41.308000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00039, (ref-fp64): 0.00085 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3092650Z E0601 05:31:41.308000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00023, (ref-fp64): 0.00046 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3097650Z E0601 05:31:41.309000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00055, (ref-fp64): 0.00130 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3102728Z E0601 05:31:41.309000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.14525, (ref-fp64): 0.11906 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3107844Z E0601 05:31:41.310000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.10787, (ref-fp64): 0.08913 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3112865Z E0601 05:31:41.310000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.10327, (ref-fp64): 0.08529 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3117881Z E0601 05:31:41.311000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00044, (ref-fp64): 0.00065 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3122971Z E0601 05:31:41.311000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00037, (ref-fp64): 0.00066 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3128186Z E0601 05:31:41.312000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02630, (ref-fp64): 0.02491 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3133077Z E0601 05:31:41.312000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00957, (ref-fp64): 0.00836 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3138364Z E0601 05:31:41.313000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00027, (ref-fp64): 0.00036 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3143165Z E0601 05:31:41.313000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00037, (ref-fp64): 0.00078 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3148276Z E0601 05:31:41.314000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00021, (ref-fp64): 0.00042 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3155505Z E0601 05:31:41.315000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00011, (ref-fp64): 0.00021 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3160802Z E0601 05:31:41.315000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00031, (ref-fp64): 0.00049 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3172334Z E0601 05:31:41.316000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00011 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3177699Z E0601 05:31:41.317000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00014, (ref-fp64): 0.00032 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3185211Z E0601 05:31:41.318000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02593, (ref-fp64): 0.02220 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3189889Z E0601 05:31:41.318000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02630, (ref-fp64): 0.02491 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3194835Z E0601 05:31:41.319000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02828, (ref-fp64): 0.02617 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3203408Z E0601 05:31:41.319000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00396, (ref-fp64): 0.00378 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3208100Z E0601 05:31:41.320000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00225, (ref-fp64): 0.00218 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3223866Z E0601 05:31:41.321000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00009, (ref-fp64): 0.00018 and shape=torch.Size([512, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3235381Z E0601 05:31:41.323000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00008 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3240229Z E0601 05:31:41.323000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00012 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3246039Z E0601 05:31:41.324000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00496, (ref-fp64): 0.00460 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3250657Z E0601 05:31:41.324000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00396, (ref-fp64): 0.00378 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3255720Z E0601 05:31:41.325000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01052, (ref-fp64): 0.01052 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3304460Z E0601 05:31:41.329000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00108, (ref-fp64): 0.00107 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3311367Z E0601 05:31:41.330000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00588, (ref-fp64): 0.00598 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3316828Z E0601 05:31:41.331000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00756, (ref-fp64): 0.00694 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3321626Z E0601 05:31:41.331000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00756, (ref-fp64): 0.00743 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3327188Z E0601 05:31:41.332000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00957, (ref-fp64): 0.00874 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3332556Z E0601 05:31:41.332000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01120, (ref-fp64): 0.01174 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3337683Z E0601 05:31:41.333000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00864, (ref-fp64): 0.00788 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3343109Z E0601 05:31:41.333000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00740, (ref-fp64): 0.00632 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3348506Z E0601 05:31:41.334000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00790, (ref-fp64): 0.00706 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3353903Z E0601 05:31:41.334000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00790, (ref-fp64): 0.00706 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3359237Z E0601 05:31:41.335000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00801, (ref-fp64): 0.00734 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3373339Z E0601 05:31:41.336000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00112, (ref-fp64): 0.00105 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3378503Z E0601 05:31:41.337000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00473, (ref-fp64): 0.00558 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3384063Z E0601 05:31:41.337000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00447, (ref-fp64): 0.00545 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3389229Z E0601 05:31:41.338000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00592, (ref-fp64): 0.00588 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3394842Z E0601 05:31:41.339000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00647, (ref-fp64): 0.01226 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3400019Z E0601 05:31:41.339000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00441, (ref-fp64): 0.00500 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3405514Z E0601 05:31:41.340000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00469, (ref-fp64): 0.00730 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3410779Z E0601 05:31:41.340000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00450, (ref-fp64): 0.00690 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3415751Z E0601 05:31:41.341000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00436, (ref-fp64): 0.00540 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3420861Z E0601 05:31:41.341000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00436, (ref-fp64): 0.00537 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3426325Z E0601 05:31:41.342000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00384, (ref-fp64): 0.00503 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3431375Z E0601 05:31:41.342000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00436, (ref-fp64): 0.00540 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3436742Z E0601 05:31:41.343000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00434, (ref-fp64): 0.00514 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3441888Z E0601 05:31:41.343000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00532, (ref-fp64): 0.00764 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3447473Z E0601 05:31:41.344000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00343, (ref-fp64): 0.00421 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3452563Z E0601 05:31:41.344000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00287, (ref-fp64): 0.00560 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3457750Z E0601 05:31:41.345000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00622, (ref-fp64): 0.00597 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3462898Z E0601 05:31:41.345000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00592, (ref-fp64): 0.00588 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3468090Z E0601 05:31:41.346000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00559, (ref-fp64): 0.01006 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3473445Z E0601 05:31:41.346000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00356, (ref-fp64): 0.00498 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3478439Z E0601 05:31:41.347000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00337, (ref-fp64): 0.00506 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3483715Z E0601 05:31:41.347000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00270, (ref-fp64): 0.00260 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3489071Z E0601 05:31:41.348000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00347, (ref-fp64): 0.01061 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3494353Z E0601 05:31:41.349000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00314, (ref-fp64): 0.00383 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3499704Z E0601 05:31:41.349000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00345, (ref-fp64): 0.00585 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3504775Z E0601 05:31:41.350000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00301, (ref-fp64): 0.00578 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3512283Z E0601 05:31:41.350000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00436, (ref-fp64): 0.00416 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3515958Z E0601 05:31:41.351000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00429, (ref-fp64): 0.00414 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3521000Z E0601 05:31:41.351000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00317, (ref-fp64): 0.00444 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3526535Z E0601 05:31:41.352000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00436, (ref-fp64): 0.00416 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3532125Z E0601 05:31:41.352000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00421, (ref-fp64): 0.00401 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3537459Z E0601 05:31:41.353000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00587, (ref-fp64): 0.00757 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3542600Z E0601 05:31:41.353000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00483, (ref-fp64): 0.00594 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3548500Z E0601 05:31:41.354000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00169, (ref-fp64): 0.00403 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3553969Z E0601 05:31:41.354000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00308, (ref-fp64): 0.00290 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3559216Z E0601 05:31:41.355000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00270, (ref-fp64): 0.00260 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3564387Z E0601 05:31:41.356000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00252, (ref-fp64): 0.00781 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3569656Z E0601 05:31:41.356000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00258, (ref-fp64): 0.00446 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3574838Z E0601 05:31:41.357000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00244, (ref-fp64): 0.00427 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3580277Z E0601 05:31:41.357000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00221, (ref-fp64): 0.00216 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3585445Z E0601 05:31:41.358000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00253, (ref-fp64): 0.00763 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3593477Z E0601 05:31:41.358000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00198, (ref-fp64): 0.00303 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3598134Z E0601 05:31:41.359000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00415, (ref-fp64): 0.00526 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3603503Z E0601 05:31:41.359000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00386, (ref-fp64): 0.00527 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3609031Z E0601 05:31:41.360000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00328, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3614368Z E0601 05:31:41.361000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00327, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3620231Z E0601 05:31:41.361000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00284, (ref-fp64): 0.00372 and shape=torch.Size([512, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3625407Z E0601 05:31:41.362000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00328, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3630869Z E0601 05:31:41.362000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00310, (ref-fp64): 0.00356 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3636367Z E0601 05:31:41.363000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00309, (ref-fp64): 0.00562 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3642012Z E0601 05:31:41.363000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00250, (ref-fp64): 0.00404 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3647949Z E0601 05:31:41.364000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00292 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3653403Z E0601 05:31:41.364000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00223, (ref-fp64): 0.00217 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3658620Z E0601 05:31:41.365000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00221, (ref-fp64): 0.00216 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3663877Z E0601 05:31:41.365000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00188, (ref-fp64): 0.00594 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3669167Z E0601 05:31:41.366000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00324, (ref-fp64): 0.00450 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3674492Z E0601 05:31:41.367000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00317, (ref-fp64): 0.00425 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3679647Z E0601 05:31:41.367000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00164 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3685141Z E0601 05:31:41.368000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00186, (ref-fp64): 0.00646 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3690531Z E0601 05:31:41.368000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00212, (ref-fp64): 0.00291 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3695782Z E0601 05:31:41.369000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00299, (ref-fp64): 0.00536 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3701100Z E0601 05:31:41.369000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00283, (ref-fp64): 0.00538 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3706213Z E0601 05:31:41.370000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00333, (ref-fp64): 0.00474 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3711713Z E0601 05:31:41.370000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00332, (ref-fp64): 0.00471 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3724064Z E0601 05:31:41.372000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00264, (ref-fp64): 0.00371 and shape=torch.Size([1024, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3729376Z E0601 05:31:41.372000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00333, (ref-fp64): 0.00474 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3735035Z E0601 05:31:41.373000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00314, (ref-fp64): 0.00448 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3741040Z E0601 05:31:41.373000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00255, (ref-fp64): 0.00538 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3745749Z E0601 05:31:41.374000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00221, (ref-fp64): 0.00417 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3752778Z E0601 05:31:41.374000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00064, (ref-fp64): 0.00237 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3764327Z E0601 05:31:41.376000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00171, (ref-fp64): 0.00167 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:31:41.3769536Z E0601 05:31:41.376000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00164 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3774870Z E0601 05:31:41.377000 140407079572096 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00103, (ref-fp64): 0.00481 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:31:41.3935075Z pass 2024-06-01T05:31:41.4002557Z TIMING: entire_frame_compile:59.04408 code_gen:23.49781 inductor_compile:38.33303 backend_compile:52.51901 2024-06-01T05:31:41.4006497Z STATS: call_* op count: 431 | FakeTensor.__torch_dispatch__:7758 | FakeTensorMode.__torch_dispatch__:49073 | attempt fast:802 | fast is_contiguous:786 | ProxyTorchDispatchMode.__torch_dispatch__:9820 | slow no contiguity match:16 2024-06-01T05:31:41.4008099Z Dynamo produced 3 graphs covering 431 ops with 7 graph breaks (5 unique) 2024-06-01T05:31:47.5213577Z 2024-06-01T05:31:49.0933709Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:31:49.0934537Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:31:49.0935033Z cuda train timm_vision_transformer 2024-06-01T05:32:42.1842819Z E0601 05:32:42.183000 139871364891264 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00049, (ref-fp64): 0.00049 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:32:42.2508372Z pass 2024-06-01T05:32:42.2686628Z TIMING: entire_frame_compile:48.50722 code_gen:10.08857 inductor_compile:25.69919 backend_compile:42.99043 2024-06-01T05:32:42.2689182Z STATS: call_* op count: 381 | FakeTensor.__torch_dispatch__:6173 | FakeTensorMode.__torch_dispatch__:49128 | attempt fast:2323 | fast is_contiguous:2323 | ProxyTorchDispatchMode.__torch_dispatch__:10943 2024-06-01T05:32:42.2690582Z Dynamo produced 2 graphs covering 381 ops with 6 graph breaks (5 unique) 2024-06-01T05:32:48.3834923Z 2024-06-01T05:33:00.8512149Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:33:00.8512784Z loading model: 0it [00:12, ?it/s] 2024-06-01T05:33:00.8513351Z cuda train timm_vision_transformer_large 2024-06-01T05:33:00.8514248Z pass_due_to_skip 2024-06-01T05:33:01.1142764Z TIMING: 2024-06-01T05:33:01.1143390Z STATS: call_* op count: 0 2024-06-01T05:33:01.1144522Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-01T05:33:04.5246446Z 2024-06-01T05:33:06.6290022Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:33:06.6290574Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:33:06.6291012Z cuda train timm_vovnet 2024-06-01T05:34:00.8932092Z E0601 05:34:00.892000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.09649, (ref-fp64): 0.09661 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.8947537Z E0601 05:34:00.894000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00398, (ref-fp64): 0.00389 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8954638Z E0601 05:34:00.894000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00236, (ref-fp64): 0.00228 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8960213Z E0601 05:34:00.895000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00069, (ref-fp64): 0.00068 and shape=torch.Size([256, 768, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8964940Z E0601 05:34:00.896000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01056, (ref-fp64): 0.01029 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8969908Z E0601 05:34:00.896000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00256, (ref-fp64): 0.00235 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8974873Z E0601 05:34:00.897000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00175, (ref-fp64): 0.00172 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.8979652Z E0601 05:34:00.897000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00915, (ref-fp64): 0.00917 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8984410Z E0601 05:34:00.898000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00243, (ref-fp64): 0.00237 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8989398Z E0601 05:34:00.898000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00153, (ref-fp64): 0.00153 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.8994374Z E0601 05:34:00.899000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00770 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.8999024Z E0601 05:34:00.899000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00180, (ref-fp64): 0.00183 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9004260Z E0601 05:34:00.899000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00136, (ref-fp64): 0.00135 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9009157Z E0601 05:34:00.900000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00614, (ref-fp64): 0.00584 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9013667Z E0601 05:34:00.900000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00149, (ref-fp64): 0.00158 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9018775Z E0601 05:34:00.901000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00101 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9023472Z E0601 05:34:00.901000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00420, (ref-fp64): 0.00408 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9030861Z E0601 05:34:00.902000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00077, (ref-fp64): 0.00074 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9037787Z E0601 05:34:00.903000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00182, (ref-fp64): 0.00178 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9042984Z E0601 05:34:00.903000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00093, (ref-fp64): 0.00091 and shape=torch.Size([512, 1056, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9047920Z E0601 05:34:00.904000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00288, (ref-fp64): 0.00280 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9052579Z E0601 05:34:00.904000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00248, (ref-fp64): 0.00229 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9057753Z E0601 05:34:00.905000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00140, (ref-fp64): 0.00137 and shape=torch.Size([160, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9062394Z E0601 05:34:00.905000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00234, (ref-fp64): 0.00220 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9067300Z E0601 05:34:00.906000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00219, (ref-fp64): 0.00203 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9072819Z E0601 05:34:00.906000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00154, (ref-fp64): 0.00146 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9077353Z E0601 05:34:00.907000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00222, (ref-fp64): 0.00210 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9081993Z E0601 05:34:00.907000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00244, (ref-fp64): 0.00230 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9087408Z E0601 05:34:00.908000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00167, (ref-fp64): 0.00158 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9091994Z E0601 05:34:00.908000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00179, (ref-fp64): 0.00171 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9096744Z E0601 05:34:00.909000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00234, (ref-fp64): 0.00214 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9101775Z E0601 05:34:00.909000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00145, (ref-fp64): 0.00140 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9108648Z E0601 05:34:00.910000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00185, (ref-fp64): 0.00187 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9114022Z E0601 05:34:00.910000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00130, (ref-fp64): 0.00126 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9120902Z E0601 05:34:00.911000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00119, (ref-fp64): 0.00118 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9128248Z E0601 05:34:00.912000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00063, (ref-fp64): 0.00063 and shape=torch.Size([768, 1472, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9134595Z E0601 05:34:00.913000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00261, (ref-fp64): 0.00256 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9140519Z E0601 05:34:00.913000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00122, (ref-fp64): 0.00119 and shape=torch.Size([192, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9147047Z E0601 05:34:00.914000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00245, (ref-fp64): 0.00234 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9152361Z E0601 05:34:00.914000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00161, (ref-fp64): 0.00160 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9158781Z E0601 05:34:00.915000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00164, (ref-fp64): 0.00167 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9163701Z E0601 05:34:00.915000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00134, (ref-fp64): 0.00132 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9170920Z E0601 05:34:00.916000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00125, (ref-fp64): 0.00126 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9175832Z E0601 05:34:00.917000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00096, (ref-fp64): 0.00096 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9182697Z E0601 05:34:00.917000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00106, (ref-fp64): 0.00107 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9187852Z E0601 05:34:00.918000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00075, (ref-fp64): 0.00075 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9200380Z E0601 05:34:00.919000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00040, (ref-fp64): 0.00041 and shape=torch.Size([768, 1728, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9207525Z E0601 05:34:00.920000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00274, (ref-fp64): 0.00274 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9215449Z E0601 05:34:00.921000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00081, (ref-fp64): 0.00080 and shape=torch.Size([192, 768, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9222097Z E0601 05:34:00.921000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00175, (ref-fp64): 0.00169 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9227154Z E0601 05:34:00.922000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00144, (ref-fp64): 0.00141 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9234187Z E0601 05:34:00.922000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00189, (ref-fp64): 0.00195 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9239126Z E0601 05:34:00.923000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00120 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9246180Z E0601 05:34:00.924000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00121 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9250937Z E0601 05:34:00.924000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00092, (ref-fp64): 0.00093 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9260099Z E0601 05:34:00.925000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00060, (ref-fp64): 0.00062 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9275398Z E0601 05:34:00.927000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00025, (ref-fp64): 0.00026 and shape=torch.Size([1024, 1888, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9289170Z E0601 05:34:00.928000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00048, (ref-fp64): 0.00049 and shape=torch.Size([224, 768, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9298326Z E0601 05:34:00.929000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00073, (ref-fp64): 0.00073 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9307494Z E0601 05:34:00.930000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00057, (ref-fp64): 0.00058 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9316796Z E0601 05:34:00.931000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00046, (ref-fp64): 0.00046 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9326109Z E0601 05:34:00.932000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00031, (ref-fp64): 0.00030 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9342454Z E0601 05:34:00.933000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00009, (ref-fp64): 0.00009 and shape=torch.Size([1024, 2144, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9361442Z E0601 05:34:00.935000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00040, (ref-fp64): 0.00041 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9370787Z E0601 05:34:00.936000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00027, (ref-fp64): 0.00027 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9386136Z E0601 05:34:00.938000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00012, (ref-fp64): 0.00012 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9390972Z E0601 05:34:00.938000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01156, (ref-fp64): 0.01108 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9395496Z E0601 05:34:00.939000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00356, (ref-fp64): 0.00327 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9400223Z E0601 05:34:00.939000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01190, (ref-fp64): 0.01186 and shape=torch.Size([64, 3, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9405045Z E0601 05:34:00.940000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01372, (ref-fp64): 0.01279 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9409873Z E0601 05:34:00.940000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00420, (ref-fp64): 0.00377 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9414633Z E0601 05:34:00.941000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00268, (ref-fp64): 0.00261 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9419249Z E0601 05:34:00.941000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01213, (ref-fp64): 0.01124 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9423758Z E0601 05:34:00.941000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00298, (ref-fp64): 0.00274 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:34:00.9428958Z E0601 05:34:00.942000 140103531967104 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00265, (ref-fp64): 0.00253 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:34:00.9969949Z pass 2024-06-01T05:34:01.0148966Z TIMING: entire_frame_compile:44.61967 code_gen:15.12012 inductor_compile:27.55931 backend_compile:39.5849 2024-06-01T05:34:01.0151974Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:4772 | FakeTensorMode.__torch_dispatch__:35991 | attempt fast:1160 | fast is_contiguous:1100 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | slow no contiguity match:60 2024-06-01T05:34:01.0153553Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-06-01T05:34:06.5157366Z 2024-06-01T05:34:08.4152038Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:34:08.4152928Z loading model: 0it [00:01, ?it/s] 2024-06-01T05:34:08.4153825Z cuda train torch_multimodal_clip 2024-06-01T05:35:11.0714111Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_302) 2024-06-01T05:35:35.8713602Z W0601 05:35:35.870000 139766165762688 torch/_logging/_internal.py:1033] [6/0] Profiler function will be ignored 2024-06-01T05:36:47.9778054Z E0601 05:36:47.977000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00616, (ref-fp64): 0.00614 and shape=torch.Size([4, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:47.9782509Z E0601 05:36:47.977000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04645, (ref-fp64): 0.04851 and shape=torch.Size([32, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0262775Z E0601 05:36:48.025000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00047, (ref-fp64): 0.00054 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0555316Z E0601 05:36:48.054000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00024, (ref-fp64): 0.00026 and shape=torch.Size([77, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0668459Z E0601 05:36:48.066000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00001, (ref-fp64): 0.00001 and shape=torch.Size([49408, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0673911Z E0601 05:36:48.066000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00338, (ref-fp64): 0.00330 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0685470Z E0601 05:36:48.068000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01129, (ref-fp64): 0.01095 and shape=torch.Size([768, 3, 32, 32]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0690418Z E0601 05:36:48.068000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00965, (ref-fp64): 0.00970 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0702897Z E0601 05:36:48.069000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01044, (ref-fp64): 0.01082 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0708405Z E0601 05:36:48.070000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.00896 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0720250Z E0601 05:36:48.071000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00914, (ref-fp64): 0.00911 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0725217Z E0601 05:36:48.072000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01015, (ref-fp64): 0.00998 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0730004Z E0601 05:36:48.072000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00954, (ref-fp64): 0.01019 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0734627Z E0601 05:36:48.073000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00865, (ref-fp64): 0.00869 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0739367Z E0601 05:36:48.073000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00966, (ref-fp64): 0.01018 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0744215Z E0601 05:36:48.074000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01467, (ref-fp64): 0.01450 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0754468Z E0601 05:36:48.075000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01307, (ref-fp64): 0.01287 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0759114Z E0601 05:36:48.075000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00937, (ref-fp64): 0.00899 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0764261Z E0601 05:36:48.076000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00893, (ref-fp64): 0.00878 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0769009Z E0601 05:36:48.076000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00926, (ref-fp64): 0.00971 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0781560Z E0601 05:36:48.077000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01049, (ref-fp64): 0.01068 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0786169Z E0601 05:36:48.078000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00952, (ref-fp64): 0.00934 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0798870Z E0601 05:36:48.079000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00935, (ref-fp64): 0.00932 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0803565Z E0601 05:36:48.079000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01053, (ref-fp64): 0.01025 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0808604Z E0601 05:36:48.080000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01076, (ref-fp64): 0.01030 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0813360Z E0601 05:36:48.080000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00999, (ref-fp64): 0.00980 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0817954Z E0601 05:36:48.081000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01080, (ref-fp64): 0.01146 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0822731Z E0601 05:36:48.081000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01543, (ref-fp64): 0.01549 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0833217Z E0601 05:36:48.082000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01410, (ref-fp64): 0.01408 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0837796Z E0601 05:36:48.083000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00968, (ref-fp64): 0.00933 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0843074Z E0601 05:36:48.083000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00897, (ref-fp64): 0.00870 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0848225Z E0601 05:36:48.084000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00642, (ref-fp64): 0.00700 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0860888Z E0601 05:36:48.085000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00715, (ref-fp64): 0.00756 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0865670Z E0601 05:36:48.086000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00460, (ref-fp64): 0.00461 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0878442Z E0601 05:36:48.087000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00555, (ref-fp64): 0.00558 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0883277Z E0601 05:36:48.087000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01110, (ref-fp64): 0.01120 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0888371Z E0601 05:36:48.088000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01205, (ref-fp64): 0.01242 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0893034Z E0601 05:36:48.088000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00416, (ref-fp64): 0.00478 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0897812Z E0601 05:36:48.089000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00579, (ref-fp64): 0.00626 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0902737Z E0601 05:36:48.089000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00869, (ref-fp64): 0.00865 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0914657Z E0601 05:36:48.090000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01148, (ref-fp64): 0.01148 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0919450Z E0601 05:36:48.091000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00452, (ref-fp64): 0.00460 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0924934Z E0601 05:36:48.092000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00476, (ref-fp64): 0.00483 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0929747Z E0601 05:36:48.092000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00443, (ref-fp64): 0.00498 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0942476Z E0601 05:36:48.093000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00473, (ref-fp64): 0.00510 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0947624Z E0601 05:36:48.094000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00058, (ref-fp64): 0.00058 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0960277Z E0601 05:36:48.095000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00182, (ref-fp64): 0.00182 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0965847Z E0601 05:36:48.096000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01060, (ref-fp64): 0.01046 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0970757Z E0601 05:36:48.096000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01211, (ref-fp64): 0.01244 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0975716Z E0601 05:36:48.097000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00345, (ref-fp64): 0.00377 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0980669Z E0601 05:36:48.097000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00379, (ref-fp64): 0.00395 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.0985862Z E0601 05:36:48.098000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00876, (ref-fp64): 0.00869 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.0995934Z E0601 05:36:48.099000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01142, (ref-fp64): 0.01138 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1001067Z E0601 05:36:48.099000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00075, (ref-fp64): 0.00076 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1006761Z E0601 05:36:48.100000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00268, (ref-fp64): 0.00267 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1011873Z E0601 05:36:48.100000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00977, (ref-fp64): 0.00978 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1024502Z E0601 05:36:48.102000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01071, (ref-fp64): 0.01081 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1029598Z E0601 05:36:48.102000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00935, (ref-fp64): 0.00932 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1042449Z E0601 05:36:48.103000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00911, (ref-fp64): 0.00910 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1047592Z E0601 05:36:48.104000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01050, (ref-fp64): 0.01087 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1052617Z E0601 05:36:48.104000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01136, (ref-fp64): 0.01109 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1057471Z E0601 05:36:48.105000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00945, (ref-fp64): 0.00929 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1062352Z E0601 05:36:48.105000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01022, (ref-fp64): 0.01008 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1067598Z E0601 05:36:48.106000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01507, (ref-fp64): 0.01503 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1077650Z E0601 05:36:48.107000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01412, (ref-fp64): 0.01409 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1082843Z E0601 05:36:48.107000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00923, (ref-fp64): 0.00922 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1088459Z E0601 05:36:48.108000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00862, (ref-fp64): 0.00860 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1093499Z E0601 05:36:48.108000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01023, (ref-fp64): 0.01039 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1106170Z E0601 05:36:48.110000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01086, (ref-fp64): 0.01098 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1111470Z E0601 05:36:48.110000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00979, (ref-fp64): 0.00967 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1123999Z E0601 05:36:48.111000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00877, (ref-fp64): 0.00880 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1129085Z E0601 05:36:48.112000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01119, (ref-fp64): 0.01105 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1133985Z E0601 05:36:48.112000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01186, (ref-fp64): 0.01176 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1139109Z E0601 05:36:48.113000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01007, (ref-fp64): 0.00969 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1144074Z E0601 05:36:48.114000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01055, (ref-fp64): 0.01049 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1149333Z E0601 05:36:48.114000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01435, (ref-fp64): 0.01435 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1159440Z E0601 05:36:48.115000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01388 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1164711Z E0601 05:36:48.116000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01011, (ref-fp64): 0.01019 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1170011Z E0601 05:36:48.116000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00913, (ref-fp64): 0.00914 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1175134Z E0601 05:36:48.117000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01050, (ref-fp64): 0.01091 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1187775Z E0601 05:36:48.118000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01100, (ref-fp64): 0.01127 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1193211Z E0601 05:36:48.118000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00959, (ref-fp64): 0.00965 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1206006Z E0601 05:36:48.120000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00850, (ref-fp64): 0.00853 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1211830Z E0601 05:36:48.120000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01125, (ref-fp64): 0.01129 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1216247Z E0601 05:36:48.121000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01149, (ref-fp64): 0.01207 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1221183Z E0601 05:36:48.121000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01076, (ref-fp64): 0.01057 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1226186Z E0601 05:36:48.122000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01141, (ref-fp64): 0.01101 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1231288Z E0601 05:36:48.122000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01285, (ref-fp64): 0.01291 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1241484Z E0601 05:36:48.123000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01341, (ref-fp64): 0.01339 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1246755Z E0601 05:36:48.124000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00910, (ref-fp64): 0.00954 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1252179Z E0601 05:36:48.124000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00833, (ref-fp64): 0.00863 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1257161Z E0601 05:36:48.125000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01239, (ref-fp64): 0.01264 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1269775Z E0601 05:36:48.126000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01182, (ref-fp64): 0.01214 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1275054Z E0601 05:36:48.127000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00926, (ref-fp64): 0.00903 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1287863Z E0601 05:36:48.128000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00821, (ref-fp64): 0.00819 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1292605Z E0601 05:36:48.128000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01060, (ref-fp64): 0.01081 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1297371Z E0601 05:36:48.129000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01175, (ref-fp64): 0.01223 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1302338Z E0601 05:36:48.129000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01347, (ref-fp64): 0.01310 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1307375Z E0601 05:36:48.130000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01296, (ref-fp64): 0.01301 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1313135Z E0601 05:36:48.130000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01121, (ref-fp64): 0.01132 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1322759Z E0601 05:36:48.131000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01288, (ref-fp64): 0.01293 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1328190Z E0601 05:36:48.132000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00965, (ref-fp64): 0.00951 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1333387Z E0601 05:36:48.132000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00876, (ref-fp64): 0.00867 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1338448Z E0601 05:36:48.133000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01286, (ref-fp64): 0.01313 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1351358Z E0601 05:36:48.134000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01206, (ref-fp64): 0.01239 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1356486Z E0601 05:36:48.135000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00939, (ref-fp64): 0.00957 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1369328Z E0601 05:36:48.136000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00844, (ref-fp64): 0.00850 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1374287Z E0601 05:36:48.137000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01113, (ref-fp64): 0.01176 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1379284Z E0601 05:36:48.137000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01201, (ref-fp64): 0.01257 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1384401Z E0601 05:36:48.138000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01299, (ref-fp64): 0.01287 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1389245Z E0601 05:36:48.138000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01300, (ref-fp64): 0.01322 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1394707Z E0601 05:36:48.139000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01071, (ref-fp64): 0.01078 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1404993Z E0601 05:36:48.140000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01260, (ref-fp64): 0.01263 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1410064Z E0601 05:36:48.140000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00941, (ref-fp64): 0.00934 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1415412Z E0601 05:36:48.141000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00857, (ref-fp64): 0.00857 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1420587Z E0601 05:36:48.141000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01292, (ref-fp64): 0.01324 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1433190Z E0601 05:36:48.142000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01210, (ref-fp64): 0.01238 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1438429Z E0601 05:36:48.143000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00841, (ref-fp64): 0.00832 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1451154Z E0601 05:36:48.144000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00782, (ref-fp64): 0.00780 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1456038Z E0601 05:36:48.145000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01234, (ref-fp64): 0.01172 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1460794Z E0601 05:36:48.145000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01322, (ref-fp64): 0.01305 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1465983Z E0601 05:36:48.146000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01136, (ref-fp64): 0.01113 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1470701Z E0601 05:36:48.146000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01183, (ref-fp64): 0.01196 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1475959Z E0601 05:36:48.147000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00977, (ref-fp64): 0.00981 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1486220Z E0601 05:36:48.148000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01239, (ref-fp64): 0.01239 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1491160Z E0601 05:36:48.148000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00837, (ref-fp64): 0.00822 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1496311Z E0601 05:36:48.149000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00769, (ref-fp64): 0.00764 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1501387Z E0601 05:36:48.149000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01111, (ref-fp64): 0.01171 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1514047Z E0601 05:36:48.150000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01083, (ref-fp64): 0.01126 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1519143Z E0601 05:36:48.151000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00731, (ref-fp64): 0.00752 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1532033Z E0601 05:36:48.152000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00714, (ref-fp64): 0.00723 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1536944Z E0601 05:36:48.153000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01153, (ref-fp64): 0.01126 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1541853Z E0601 05:36:48.153000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01321, (ref-fp64): 0.01309 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1547051Z E0601 05:36:48.154000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00869, (ref-fp64): 0.00883 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1552326Z E0601 05:36:48.154000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00992, (ref-fp64): 0.01026 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1557604Z E0601 05:36:48.155000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00892, (ref-fp64): 0.00888 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1567691Z E0601 05:36:48.156000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01193, (ref-fp64): 0.01196 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1572606Z E0601 05:36:48.156000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00746, (ref-fp64): 0.00738 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1577821Z E0601 05:36:48.157000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00711, (ref-fp64): 0.00704 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1582939Z E0601 05:36:48.157000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00891, (ref-fp64): 0.00946 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1595580Z E0601 05:36:48.159000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00922, (ref-fp64): 0.00965 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1600448Z E0601 05:36:48.159000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00647, (ref-fp64): 0.00654 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1613694Z E0601 05:36:48.160000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00640, (ref-fp64): 0.00643 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1618566Z E0601 05:36:48.161000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01201, (ref-fp64): 0.01146 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1623429Z E0601 05:36:48.161000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01335, (ref-fp64): 0.01301 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1628670Z E0601 05:36:48.162000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00658, (ref-fp64): 0.00658 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1633728Z E0601 05:36:48.162000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00820, (ref-fp64): 0.00844 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1638963Z E0601 05:36:48.163000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00875, (ref-fp64): 0.00878 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1649085Z E0601 05:36:48.164000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01170, (ref-fp64): 0.01173 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1655676Z E0601 05:36:48.165000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00689, (ref-fp64): 0.00687 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1660922Z E0601 05:36:48.165000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00666, (ref-fp64): 0.00668 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1665940Z E0601 05:36:48.166000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00061, (ref-fp64): 0.00053 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1671099Z E0601 05:36:48.166000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00272, (ref-fp64): 0.00259 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1676188Z E0601 05:36:48.167000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00939, (ref-fp64): 0.00920 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1681067Z E0601 05:36:48.167000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00988, (ref-fp64): 0.00910 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1686739Z E0601 05:36:48.168000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01120, (ref-fp64): 0.01107 and shape=torch.Size([50, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1691872Z E0601 05:36:48.168000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00150, (ref-fp64): 0.00157 and shape=torch.Size([768, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1697109Z E0601 05:36:48.169000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00669, (ref-fp64): 0.00763 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1703843Z E0601 05:36:48.169000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00783, (ref-fp64): 0.00906 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1709077Z E0601 05:36:48.170000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00584, (ref-fp64): 0.00648 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1716533Z E0601 05:36:48.171000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00695, (ref-fp64): 0.00799 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1720814Z E0601 05:36:48.171000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00625, (ref-fp64): 0.00792 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1726082Z E0601 05:36:48.172000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00703, (ref-fp64): 0.00841 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1731172Z E0601 05:36:48.172000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00643, (ref-fp64): 0.00750 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1736173Z E0601 05:36:48.173000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00789, (ref-fp64): 0.00972 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1741400Z E0601 05:36:48.173000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00919, (ref-fp64): 0.00947 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1747329Z E0601 05:36:48.174000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00907, (ref-fp64): 0.01007 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1752594Z E0601 05:36:48.174000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00562, (ref-fp64): 0.00695 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1757857Z E0601 05:36:48.175000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00690, (ref-fp64): 0.00816 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1763192Z E0601 05:36:48.175000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00639, (ref-fp64): 0.00811 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1769915Z E0601 05:36:48.176000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00785, (ref-fp64): 0.00934 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1775038Z E0601 05:36:48.177000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00525, (ref-fp64): 0.00621 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1781594Z E0601 05:36:48.177000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00676, (ref-fp64): 0.00766 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1786808Z E0601 05:36:48.178000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00674, (ref-fp64): 0.00682 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1791742Z E0601 05:36:48.178000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00788, (ref-fp64): 0.00807 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1796907Z E0601 05:36:48.179000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00622, (ref-fp64): 0.00720 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1801747Z E0601 05:36:48.179000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00722, (ref-fp64): 0.00890 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1807359Z E0601 05:36:48.180000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00878, (ref-fp64): 0.00888 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1813155Z E0601 05:36:48.180000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00878, (ref-fp64): 0.00904 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1818279Z E0601 05:36:48.181000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00593, (ref-fp64): 0.00662 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1823499Z E0601 05:36:48.181000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00663, (ref-fp64): 0.00763 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1828676Z E0601 05:36:48.182000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00684, (ref-fp64): 0.00779 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1835615Z E0601 05:36:48.183000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00854, (ref-fp64): 0.00941 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1840457Z E0601 05:36:48.183000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00271, (ref-fp64): 0.00282 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1847470Z E0601 05:36:48.184000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00585, (ref-fp64): 0.00611 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1852321Z E0601 05:36:48.184000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00479, (ref-fp64): 0.00540 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1857204Z E0601 05:36:48.185000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00695, (ref-fp64): 0.00706 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1862135Z E0601 05:36:48.185000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00499, (ref-fp64): 0.00546 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1867029Z E0601 05:36:48.186000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00824, (ref-fp64): 0.00904 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1872538Z E0601 05:36:48.186000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00819, (ref-fp64): 0.00817 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1878316Z E0601 05:36:48.187000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01012, (ref-fp64): 0.01024 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1883380Z E0601 05:36:48.187000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00328, (ref-fp64): 0.00398 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1888817Z E0601 05:36:48.188000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00537, (ref-fp64): 0.00567 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1893949Z E0601 05:36:48.188000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00683, (ref-fp64): 0.00756 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1900679Z E0601 05:36:48.189000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00840, (ref-fp64): 0.00918 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1905788Z E0601 05:36:48.190000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00192, (ref-fp64): 0.00215 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1912715Z E0601 05:36:48.190000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00613, (ref-fp64): 0.00640 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1917766Z E0601 05:36:48.191000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00393, (ref-fp64): 0.00453 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1922628Z E0601 05:36:48.191000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00733, (ref-fp64): 0.00787 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1928042Z E0601 05:36:48.192000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00686, (ref-fp64): 0.00718 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1932846Z E0601 05:36:48.192000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00806, (ref-fp64): 0.00893 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1937915Z E0601 05:36:48.193000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00794, (ref-fp64): 0.00863 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1943638Z E0601 05:36:48.193000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01080, (ref-fp64): 0.01110 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1948825Z E0601 05:36:48.194000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00185, (ref-fp64): 0.00256 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1954179Z E0601 05:36:48.195000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00520, (ref-fp64): 0.00524 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1959144Z E0601 05:36:48.195000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00672, (ref-fp64): 0.00850 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1966233Z E0601 05:36:48.196000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00811, (ref-fp64): 0.00963 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1971275Z E0601 05:36:48.196000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00445, (ref-fp64): 0.00593 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1977780Z E0601 05:36:48.197000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00633, (ref-fp64): 0.00746 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.1982735Z E0601 05:36:48.197000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00549, (ref-fp64): 0.00654 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1987533Z E0601 05:36:48.198000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00668, (ref-fp64): 0.00810 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1992633Z E0601 05:36:48.198000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00742, (ref-fp64): 0.00847 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.1997467Z E0601 05:36:48.199000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00779, (ref-fp64): 0.00879 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2002784Z E0601 05:36:48.199000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00869, (ref-fp64): 0.00868 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2008981Z E0601 05:36:48.200000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00920, (ref-fp64): 0.00947 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2013876Z E0601 05:36:48.200000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00533, (ref-fp64): 0.00630 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2019127Z E0601 05:36:48.201000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00638, (ref-fp64): 0.00734 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2024141Z E0601 05:36:48.202000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00694, (ref-fp64): 0.00932 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2030925Z E0601 05:36:48.202000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00801, (ref-fp64): 0.01002 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2036067Z E0601 05:36:48.203000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00386, (ref-fp64): 0.00614 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2042721Z E0601 05:36:48.203000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00618, (ref-fp64): 0.00744 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2047988Z E0601 05:36:48.204000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00543, (ref-fp64): 0.00625 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2052826Z E0601 05:36:48.204000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00752, (ref-fp64): 0.00779 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2057978Z E0601 05:36:48.205000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00552, (ref-fp64): 0.00816 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2062846Z E0601 05:36:48.205000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00726, (ref-fp64): 0.00919 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2068175Z E0601 05:36:48.206000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00884, (ref-fp64): 0.00915 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2074444Z E0601 05:36:48.207000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01008, (ref-fp64): 0.01040 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2079444Z E0601 05:36:48.207000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00444, (ref-fp64): 0.00550 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2084973Z E0601 05:36:48.208000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00615, (ref-fp64): 0.00686 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2090092Z E0601 05:36:48.208000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00706, (ref-fp64): 0.00873 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2096767Z E0601 05:36:48.209000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00804, (ref-fp64): 0.00977 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2101856Z E0601 05:36:48.209000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00440, (ref-fp64): 0.00609 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2108572Z E0601 05:36:48.210000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00609, (ref-fp64): 0.00718 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2113785Z E0601 05:36:48.210000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00652, (ref-fp64): 0.00761 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2118730Z E0601 05:36:48.211000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00772, (ref-fp64): 0.00809 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2123785Z E0601 05:36:48.211000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00552, (ref-fp64): 0.00675 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2128898Z E0601 05:36:48.212000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00642, (ref-fp64): 0.00763 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2134095Z E0601 05:36:48.212000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00857, (ref-fp64): 0.00997 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2140044Z E0601 05:36:48.213000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00968, (ref-fp64): 0.01080 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2145107Z E0601 05:36:48.214000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00469, (ref-fp64): 0.00624 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2150545Z E0601 05:36:48.214000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00580, (ref-fp64): 0.00715 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2155773Z E0601 05:36:48.215000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00708, (ref-fp64): 0.00819 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2162334Z E0601 05:36:48.215000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00834, (ref-fp64): 0.00953 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2168010Z E0601 05:36:48.216000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00449, (ref-fp64): 0.00478 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2174496Z E0601 05:36:48.217000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00588, (ref-fp64): 0.00652 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2179668Z E0601 05:36:48.217000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00615, (ref-fp64): 0.00651 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2184596Z E0601 05:36:48.218000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00754, (ref-fp64): 0.00790 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2189735Z E0601 05:36:48.218000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00635, (ref-fp64): 0.00764 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2194821Z E0601 05:36:48.219000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00784, (ref-fp64): 0.00940 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2199879Z E0601 05:36:48.219000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00886, (ref-fp64): 0.00871 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2206142Z E0601 05:36:48.220000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01024, (ref-fp64): 0.01044 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2211280Z E0601 05:36:48.220000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00450, (ref-fp64): 0.00501 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2216365Z E0601 05:36:48.221000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00562, (ref-fp64): 0.00627 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2221623Z E0601 05:36:48.221000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00674, (ref-fp64): 0.00799 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2228313Z E0601 05:36:48.222000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00814, (ref-fp64): 0.00950 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2233307Z E0601 05:36:48.222000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00416, (ref-fp64): 0.00452 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2240272Z E0601 05:36:48.223000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00579, (ref-fp64): 0.00639 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2245620Z E0601 05:36:48.224000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00585, (ref-fp64): 0.00723 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2250405Z E0601 05:36:48.224000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00759, (ref-fp64): 0.00883 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2255535Z E0601 05:36:48.225000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00611, (ref-fp64): 0.00736 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2260421Z E0601 05:36:48.225000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00722, (ref-fp64): 0.00919 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2265704Z E0601 05:36:48.226000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00843, (ref-fp64): 0.00892 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2271602Z E0601 05:36:48.226000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01030, (ref-fp64): 0.01079 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2276934Z E0601 05:36:48.227000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00420, (ref-fp64): 0.00463 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2281889Z E0601 05:36:48.227000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00547, (ref-fp64): 0.00573 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2287358Z E0601 05:36:48.228000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00697, (ref-fp64): 0.00819 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2294131Z E0601 05:36:48.228000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00825, (ref-fp64): 0.00958 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2299213Z E0601 05:36:48.229000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00374, (ref-fp64): 0.00427 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2305907Z E0601 05:36:48.230000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00585, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2311290Z E0601 05:36:48.230000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00617, (ref-fp64): 0.00748 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2316126Z E0601 05:36:48.231000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00788, (ref-fp64): 0.00869 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2321091Z E0601 05:36:48.231000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00619, (ref-fp64): 0.00794 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2326414Z E0601 05:36:48.232000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00764, (ref-fp64): 0.00780 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2331308Z E0601 05:36:48.232000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00865, (ref-fp64): 0.00880 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2337136Z E0601 05:36:48.233000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01011, (ref-fp64): 0.01047 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2342268Z E0601 05:36:48.233000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00373, (ref-fp64): 0.00418 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2347355Z E0601 05:36:48.234000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00529, (ref-fp64): 0.00581 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2352560Z E0601 05:36:48.234000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00692, (ref-fp64): 0.00878 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2360970Z E0601 05:36:48.235000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00815, (ref-fp64): 0.00961 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2364490Z E0601 05:36:48.236000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00318, (ref-fp64): 0.00323 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2371273Z E0601 05:36:48.236000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00583, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2376348Z E0601 05:36:48.237000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00544, (ref-fp64): 0.00630 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2381349Z E0601 05:36:48.237000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00690, (ref-fp64): 0.00746 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2386236Z E0601 05:36:48.238000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00477, (ref-fp64): 0.00583 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2391447Z E0601 05:36:48.238000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00709, (ref-fp64): 0.00807 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2396355Z E0601 05:36:48.239000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00781, (ref-fp64): 0.00856 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2402307Z E0601 05:36:48.239000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01008, (ref-fp64): 0.01051 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2407697Z E0601 05:36:48.240000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00329, (ref-fp64): 0.00439 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2412811Z E0601 05:36:48.240000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00521, (ref-fp64): 0.00583 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2417812Z E0601 05:36:48.241000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00654, (ref-fp64): 0.00763 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2424550Z E0601 05:36:48.242000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00802, (ref-fp64): 0.00921 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2429571Z E0601 05:36:48.242000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00371, (ref-fp64): 0.00398 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2436343Z E0601 05:36:48.243000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00593, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2441253Z E0601 05:36:48.243000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00474, (ref-fp64): 0.00635 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2446510Z E0601 05:36:48.244000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00628, (ref-fp64): 0.00749 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2451488Z E0601 05:36:48.244000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00527, (ref-fp64): 0.00590 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2456376Z E0601 05:36:48.245000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00790, (ref-fp64): 0.00894 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2461579Z E0601 05:36:48.245000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00880, (ref-fp64): 0.00907 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2467361Z E0601 05:36:48.246000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01103, (ref-fp64): 0.01121 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2472550Z E0601 05:36:48.246000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00279, (ref-fp64): 0.00327 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2477839Z E0601 05:36:48.247000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00542, (ref-fp64): 0.00547 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2482970Z E0601 05:36:48.247000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00084 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2488123Z E0601 05:36:48.248000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00697, (ref-fp64): 0.00716 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:36:48.2493296Z E0601 05:36:48.248000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00145, (ref-fp64): 0.00174 and shape=torch.Size([77, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2498488Z E0601 05:36:48.249000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00443, (ref-fp64): 0.00467 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2608340Z E0601 05:36:48.260000 139766165762688 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00008, (ref-fp64): 0.00009 and shape=torch.Size([49408, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:36:48.2654353Z pass 2024-06-01T05:36:48.3476369Z TIMING: entire_frame_compile:149.91474 code_gen:41.71859 inductor_compile:84.45326 backend_compile:131.6399 2024-06-01T05:36:48.3478180Z STATS: call_* op count: 963 | FakeTensor.__torch_dispatch__:25505 | FakeTensorMode.__torch_dispatch__:178520 | attempt fast:2221 | fast is_contiguous:2139 | slow no contiguity match:82 | ProxyTorchDispatchMode.__torch_dispatch__:40036 2024-06-01T05:36:48.3480339Z Dynamo produced 3 graphs covering 963 ops with 7 graph breaks (5 unique) 2024-06-01T05:36:59.5070240Z 2024-06-01T05:37:00.1050453Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:37:00.1050935Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:37:00.1051378Z cuda train tts_angular 2024-06-01T05:37:04.0352173Z W0601 05:37:04.034000 139835843211904 torch/_logging/_internal.py:1033] [10/0] Profiler function will be ignored 2024-06-01T05:37:10.9037635Z E0601 05:37:10.902000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00068, (ref-fp64): 0.00070 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9042795Z E0601 05:37:10.903000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00163, (ref-fp64): 0.00172 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9047809Z E0601 05:37:10.904000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00163, (ref-fp64): 0.00172 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9060232Z E0601 05:37:10.905000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00226, (ref-fp64): 0.00226 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9066042Z E0601 05:37:10.906000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00141, (ref-fp64): 0.00142 and shape=torch.Size([3072, 40]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9071540Z E0601 05:37:10.906000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00069, (ref-fp64): 0.00071 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9076655Z E0601 05:37:10.907000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00150, (ref-fp64): 0.00151 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9081558Z E0601 05:37:10.907000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00150, (ref-fp64): 0.00151 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9094185Z E0601 05:37:10.909000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00330, (ref-fp64): 0.00330 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9100023Z E0601 05:37:10.909000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00265, (ref-fp64): 0.00265 and shape=torch.Size([3072, 256]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9105266Z E0601 05:37:10.910000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00056, (ref-fp64): 0.00056 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9110435Z E0601 05:37:10.910000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00130, (ref-fp64): 0.00135 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9115543Z E0601 05:37:10.911000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00130, (ref-fp64): 0.00135 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9128213Z E0601 05:37:10.912000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00442, (ref-fp64): 0.00441 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9134248Z E0601 05:37:10.913000 139835843211904 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00342, (ref-fp64): 0.00341 and shape=torch.Size([3072, 256]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:10.9139947Z pass 2024-06-01T05:37:10.9206893Z TIMING: entire_frame_compile:7.96003 inductor_compile:6.5713 backend_compile:7.17547 code_gen:5.59549 2024-06-01T05:37:10.9208228Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4838 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:860 2024-06-01T05:37:10.9209400Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-06-01T05:37:14.6584566Z 2024-06-01T05:37:17.0359740Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:37:17.0360228Z loading model: 0it [00:02, ?it/s] 2024-06-01T05:37:17.0360665Z cuda train vgg16 2024-06-01T05:37:55.4540073Z E0601 05:37:55.453000 139762153177728 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01018, (ref-fp64): 0.00945 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:37:55.5221606Z pass 2024-06-01T05:37:55.5434042Z TIMING: entire_frame_compile:12.83093 code_gen:6.56856 inductor_compile:9.50817 backend_compile:12.16215 2024-06-01T05:37:55.5436266Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:808 | FakeTensorMode.__torch_dispatch__:7643 | attempt fast:36 | fast is_contiguous:36 | ProxyTorchDispatchMode.__torch_dispatch__:1686 2024-06-01T05:37:55.5438262Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-06-01T05:37:59.5325033Z 2024-06-01T05:38:02.8443080Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:38:02.8443563Z loading model: 0it [00:03, ?it/s] 2024-06-01T05:38:02.8447530Z cuda train vision_maskrcnn 2024-06-01T05:38:34.3311732Z W0601 05:38:34.330000 140134850790016 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:38:34.3319835Z W0601 05:38:34.331000 140134850790016 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:38:34.3330204Z W0601 05:38:34.332000 140134850790016 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:38:34.3339971Z W0601 05:38:34.333000 140134850790016 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:38:34.3351658Z W0601 05:38:34.334000 140134850790016 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program 2024-06-01T05:38:36.2284913Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_7). Found from : 2024-06-01T05:38:36.2287396Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward 2024-06-01T05:38:36.2288386Z anchors = self.anchor_generator(images, features) 2024-06-01T05:38:36.2289404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:38:36.2290267Z return forward_call(*args, **kwargs) 2024-06-01T05:38:36.2291291Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward 2024-06-01T05:38:36.2292597Z self.set_cell_anchors(dtype, device) 2024-06-01T05:38:36.2293966Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors 2024-06-01T05:38:36.2295260Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-01T05:38:36.2296594Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in 2024-06-01T05:38:36.2297821Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-01T05:38:36.2298420Z 2024-06-01T05:38:37.6818099Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-06-01T05:38:37.6819996Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-01T05:38:37.6821405Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] or: 2024-06-01T05:38:37.6822532Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-01T05:38:37.6823912Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] to include these operations in the captured graph. 2024-06-01T05:38:37.6825018Z W0601 05:38:37.681000 140134850790016 torch/_dynamo/variables/tensor.py:715] [18/0] 2024-06-01T05:38:49.9134019Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:38:49.9135975Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:38:49.9137152Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:38:49.9137773Z 2024-06-01T05:38:56.4919048Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:38:56.4920497Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:38:56.4921703Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:38:56.4922462Z 2024-06-01T05:39:02.5777433Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:39:02.5778736Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:39:02.5779939Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:39:02.5780556Z 2024-06-01T05:39:06.9951635Z W0601 05:39:06.993000 140134850790016 torch/fx/experimental/symbolic_shapes.py:4455] [9/1] RecursionError in sympy.xreplace(Ne(Mod(2*(((s4 + 1)//2)), s4), 0), {s4: evaluate_static_shape_0 + 3}) 2024-06-01T05:39:09.7932145Z W0601 05:39:09.791000 140134850790016 torch/fx/experimental/symbolic_shapes.py:4455] [9/1] RecursionError in sympy.xreplace(Eq(Mod(2*(((s4 + 1)//2)), s4), 0), {s4: evaluate_static_shape_0 + 3}) 2024-06-01T05:39:39.6837300Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:39:39.6839950Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:39:39.6842012Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:39:39.6843001Z 2024-06-01T05:39:49.4563260Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:39:49.4564741Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:39:49.4567163Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:39:49.4567804Z 2024-06-01T05:40:03.5693277Z W0601 05:40:03.568000 140134850790016 torch/fx/experimental/symbolic_shapes.py:4455] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-06-01T05:40:05.5945394Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:05.5947057Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:40:05.5948422Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:40:05.5949040Z 2024-06-01T05:40:11.5671979Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:11.5674364Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:40:11.5675583Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:40:11.5676204Z 2024-06-01T05:40:18.7309518Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:18.7311589Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-06-01T05:40:18.7313043Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-06-01T05:40:18.7313599Z 2024-06-01T05:40:19.6614505Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:19.6616149Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-06-01T05:40:19.6617531Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-06-01T05:40:19.6618024Z 2024-06-01T05:40:20.7003766Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:20.7005316Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804 2024-06-01T05:40:20.7006527Z masks_probs = maskrcnn_inference(mask_logits, labels) 2024-06-01T05:40:20.7007769Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference 2024-06-01T05:40:20.7008800Z mask_prob = mask_prob[index, labels][:, None] 2024-06-01T05:40:20.7009147Z 2024-06-01T05:40:22.7121754Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:22.7123229Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-01T05:40:22.7124402Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-01T05:40:22.7125002Z 2024-06-01T05:40:26.3164604Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:26.3166252Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward 2024-06-01T05:40:26.3167574Z features = self.backbone(images.tensors) 2024-06-01T05:40:26.3169351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:40:26.3170368Z return forward_call(*args, **kwargs) 2024-06-01T05:40:26.3171689Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward 2024-06-01T05:40:26.3172934Z x = self.fpn(x) 2024-06-01T05:40:26.3173792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1549, in _call_impl 2024-06-01T05:40:26.3174666Z return forward_call(*args, **kwargs) 2024-06-01T05:40:26.3175685Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward 2024-06-01T05:40:26.3176789Z inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest") 2024-06-01T05:40:26.3177276Z 2024-06-01T05:40:31.1513687Z W0601 05:40:31.150000 140134850790016 torch/_dynamo/convert_frame.py:744] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-06-01T05:40:31.1515625Z W0601 05:40:31.150000 140134850790016 torch/_dynamo/convert_frame.py:744] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-06-01T05:40:31.1517525Z W0601 05:40:31.150000 140134850790016 torch/_dynamo/convert_frame.py:744] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623 2024-06-01T05:40:31.1519077Z W0601 05:40:31.150000 140134850790016 torch/_dynamo/convert_frame.py:744] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-06-01T05:40:31.1520755Z W0601 05:40:31.150000 140134850790016 torch/_dynamo/convert_frame.py:744] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-06-01T05:40:39.4910655Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:39.4912290Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:40:39.4913486Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:40:39.4914772Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:40:39.4915680Z v1 = masked_index(y_low, x_low) 2024-06-01T05:40:39.4916605Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:40:39.4917435Z return input[ 2024-06-01T05:40:39.4917632Z 2024-06-01T05:40:45.5363577Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:45.5365380Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:40:45.5367073Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:40:45.5368939Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:40:45.5370236Z v1 = masked_index(y_low, x_low) 2024-06-01T05:40:45.5371507Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:40:45.5372666Z return input[ 2024-06-01T05:40:45.5372959Z 2024-06-01T05:40:53.1584279Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:40:53.1585829Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:40:53.1587122Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:40:53.1588413Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:40:53.1589617Z v1 = masked_index(y_low, x_low) 2024-06-01T05:40:53.1590845Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:40:53.1591693Z return input[ 2024-06-01T05:40:53.1591901Z 2024-06-01T05:41:08.1158808Z W0601 05:41:08.114000 140134850790016 torch/fx/experimental/symbolic_shapes.py:4455] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-06-01T05:41:09.3134185Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:41:09.3135628Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:41:09.3136763Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:41:09.3138057Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:41:09.3139231Z v1 = masked_index(y_low, x_low) 2024-06-01T05:41:09.3140180Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:41:09.3141015Z return input[ 2024-06-01T05:41:09.3141223Z 2024-06-01T05:41:18.5766248Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:41:18.5767617Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:41:18.5768751Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:41:18.5770034Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:41:18.5770988Z v1 = masked_index(y_low, x_low) 2024-06-01T05:41:18.5771915Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:41:18.5772782Z return input[ 2024-06-01T05:41:18.5772982Z 2024-06-01T05:41:27.6234894Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:41:27.6236381Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:41:27.6237528Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:41:27.6238796Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:41:27.6239701Z v1 = masked_index(y_low, x_low) 2024-06-01T05:41:27.6240621Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:41:27.6241442Z return input[ 2024-06-01T05:41:27.6241646Z 2024-06-01T05:41:35.5512695Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:41:35.5515034Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:41:35.5516182Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:41:35.5517460Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:41:35.5518373Z v1 = masked_index(y_low, x_low) 2024-06-01T05:41:35.5519304Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:41:35.5520132Z return input[ 2024-06-01T05:41:35.5520340Z 2024-06-01T05:41:36.6585071Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-01T05:41:36.6586575Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-01T05:41:36.6587749Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-01T05:41:36.6589301Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-01T05:41:36.6590529Z v1 = masked_index(y_low, x_low) 2024-06-01T05:41:36.6591470Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-01T05:41:36.6592315Z return input[ 2024-06-01T05:41:36.6592514Z 2024-06-01T05:41:36.8857836Z E0601 05:41:36.885000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.06520, (ref-fp64): 0.24607 and shape=torch.Size([4, 4]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8868085Z E0601 05:41:36.886000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00232, (ref-fp64): 0.00435 and shape=torch.Size([4, 1, 427, 640]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8901638Z E0601 05:41:36.889000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00219, (ref-fp64): 0.00267 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8906072Z E0601 05:41:36.890000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00093, (ref-fp64): 0.00123 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8910981Z E0601 05:41:36.890000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00121, (ref-fp64): 0.00157 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8918049Z E0601 05:41:36.891000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00062, (ref-fp64): 0.00072 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8922894Z E0601 05:41:36.891000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00092, (ref-fp64): 0.00098 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8927690Z E0601 05:41:36.892000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00116 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8932461Z E0601 05:41:36.892000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00097, (ref-fp64): 0.00121 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8937405Z E0601 05:41:36.893000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00098, (ref-fp64): 0.00113 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8941993Z E0601 05:41:36.893000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00096, (ref-fp64): 0.00117 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8946725Z E0601 05:41:36.894000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00139 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8951776Z E0601 05:41:36.894000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00139 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8956484Z E0601 05:41:36.895000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00146 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8961194Z E0601 05:41:36.895000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00114, (ref-fp64): 0.00134 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8966196Z E0601 05:41:36.896000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00120, (ref-fp64): 0.00139 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8971041Z E0601 05:41:36.896000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00073, (ref-fp64): 0.00083 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8975513Z E0601 05:41:36.897000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00099, (ref-fp64): 0.00109 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8980334Z E0601 05:41:36.897000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00068, (ref-fp64): 0.00074 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8985114Z E0601 05:41:36.898000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00061, (ref-fp64): 0.00070 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8989885Z E0601 05:41:36.898000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00076, (ref-fp64): 0.00086 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.8994587Z E0601 05:41:36.899000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00096, (ref-fp64): 0.00107 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.8999233Z E0601 05:41:36.899000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00086, (ref-fp64): 0.00093 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9004362Z E0601 05:41:36.900000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00094, (ref-fp64): 0.00098 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9009110Z E0601 05:41:36.900000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00106, (ref-fp64): 0.00114 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9013597Z E0601 05:41:36.900000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00108 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9018469Z E0601 05:41:36.901000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00096, (ref-fp64): 0.00100 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9023101Z E0601 05:41:36.901000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00110 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9027700Z E0601 05:41:36.902000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00116, (ref-fp64): 0.00121 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9032529Z E0601 05:41:36.902000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00101, (ref-fp64): 0.00104 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9037237Z E0601 05:41:36.903000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00101, (ref-fp64): 0.00108 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9041822Z E0601 05:41:36.903000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00109, (ref-fp64): 0.00115 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9047044Z E0601 05:41:36.904000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00095, (ref-fp64): 0.00103 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9051513Z E0601 05:41:36.904000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00099, (ref-fp64): 0.00110 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9056177Z E0601 05:41:36.905000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00094, (ref-fp64): 0.00102 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9068717Z E0601 05:41:36.906000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00063, (ref-fp64): 0.00070 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9075837Z E0601 05:41:36.907000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00082, (ref-fp64): 0.00094 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9087170Z E0601 05:41:36.908000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00086, (ref-fp64): 0.00100 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9093435Z E0601 05:41:36.908000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00080, (ref-fp64): 0.00092 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9105966Z E0601 05:41:36.910000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00078, (ref-fp64): 0.00089 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9112406Z E0601 05:41:36.910000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00096, (ref-fp64): 0.00103 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9118940Z E0601 05:41:36.911000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00115 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9131589Z E0601 05:41:36.912000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00080, (ref-fp64): 0.00096 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9137757Z E0601 05:41:36.913000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00087, (ref-fp64): 0.00102 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9142067Z E0601 05:41:36.913000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00005, (ref-fp64): 0.00077 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9146903Z E0601 05:41:36.914000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00001, (ref-fp64): 0.00017 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9151284Z E0601 05:41:36.914000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00005, (ref-fp64): 0.00078 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9155934Z E0601 05:41:36.915000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00030 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9160062Z E0601 05:41:36.915000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00186 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9165107Z E0601 05:41:36.916000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00018, (ref-fp64): 0.00035 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9169370Z E0601 05:41:36.916000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00104, (ref-fp64): 0.00186 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9173997Z E0601 05:41:36.916000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00063, (ref-fp64): 0.00093 and shape=torch.Size([256, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9178362Z E0601 05:41:36.917000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00057 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9183007Z E0601 05:41:36.917000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00003, (ref-fp64): 0.00033 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9191811Z E0601 05:41:36.918000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00079, (ref-fp64): 0.00118 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9196807Z E0601 05:41:36.919000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00075, (ref-fp64): 0.00100 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9205605Z E0601 05:41:36.920000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00036, (ref-fp64): 0.00047 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9262435Z E0601 05:41:36.925000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00071, (ref-fp64): 0.00083 and shape=torch.Size([1024, 12544]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9266641Z E0601 05:41:36.926000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00022, (ref-fp64): 0.00066 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9272899Z E0601 05:41:36.926000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00074, (ref-fp64): 0.00200 and shape=torch.Size([1024, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9277266Z E0601 05:41:36.927000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00005, (ref-fp64): 0.00034 and shape=torch.Size([364]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:41:36.9281779Z E0601 05:41:36.927000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00056, (ref-fp64): 0.00087 and shape=torch.Size([364, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:36.9536104Z E0601 05:41:36.953000 140134850790016 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00001, (ref-fp64): 0.00002 and shape=torch.Size([1024, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:41:37.0053455Z pass 2024-06-01T05:41:37.0627597Z TIMING: entire_frame_compile:195.19047 inductor_compile:94.75923 backend_compile:152.44149 code_gen:46.9929 2024-06-01T05:41:37.0629576Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:186096 | FakeTensor.__torch_dispatch__:14875 | ProxyTorchDispatchMode.__torch_dispatch__:45365 | attempt fast:3042 | slow no contiguity match:290 | fast is_contiguous:2100 | slow both tensors nontrivially broadcast:652 2024-06-01T05:41:37.0631563Z Dynamo produced 51 graphs covering 2515 ops with 34 graph breaks (8 unique) 2024-06-01T05:41:48.5098742Z 2024-06-01T05:41:51.5143512Z loading model: 0it [00:00, ?it/s] 2024-06-01T05:41:51.5144388Z loading model: 0it [00:03, ?it/s] 2024-06-01T05:41:51.5145242Z cuda train yolov3 2024-06-01T05:42:35.4846364Z W0601 05:42:35.484000 139828816900736 torch/_inductor/utils.py:1189] [9/0] DeviceCopy in input program 2024-06-01T05:42:35.4850786Z W0601 05:42:35.484000 139828816900736 torch/_inductor/utils.py:1189] [9/0] DeviceCopy in input program 2024-06-01T05:42:36.5201500Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_2). Found from : 2024-06-01T05:42:36.5202993Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-01T05:42:36.5204799Z self.create_grids((nx, ny), p.device) 2024-06-01T05:42:36.5206762Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-01T05:42:36.5208154Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-01T05:42:36.5208661Z 2024-06-01T05:42:36.8541966Z W0601 05:42:36.853000 139828816900736 torch/_inductor/utils.py:1189] [9/1] DeviceCopy in input program 2024-06-01T05:42:36.8547096Z W0601 05:42:36.854000 139828816900736 torch/_inductor/utils.py:1189] [9/1] DeviceCopy in input program 2024-06-01T05:42:38.0685172Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_4). Found from : 2024-06-01T05:42:38.0686729Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-01T05:42:38.0687881Z self.create_grids((nx, ny), p.device) 2024-06-01T05:42:38.0688976Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-01T05:42:38.0690000Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-01T05:42:38.0690352Z 2024-06-01T05:42:38.6183200Z W0601 05:42:38.617000 139828816900736 torch/_logging/_internal.py:1033] [13/0] Profiler function will be ignored 2024-06-01T05:43:55.4148830Z E0601 05:43:55.414000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.37249, (ref-fp64): 1.37061 and shape=torch.Size([4, 3, 12, 16, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4153096Z E0601 05:43:55.414000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 1.09289, (ref-fp64): 1.09266 and shape=torch.Size([4, 3, 24, 32, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4165323Z E0601 05:43:55.416000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.80667, (ref-fp64): 0.80650 and shape=torch.Size([4, 3, 48, 64, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4173522Z E0601 05:43:55.416000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03739, (ref-fp64): 0.03662 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4177615Z E0601 05:43:55.417000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04521, (ref-fp64): 0.04498 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4182169Z E0601 05:43:55.417000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.07105, (ref-fp64): 0.07055 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4186308Z E0601 05:43:55.418000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01390, (ref-fp64): 0.01372 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4190994Z E0601 05:43:55.418000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01912, (ref-fp64): 0.01913 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4195267Z E0601 05:43:55.419000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.04017, (ref-fp64): 0.04008 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4201306Z E0601 05:43:55.419000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00618, (ref-fp64): 0.00618 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4206338Z E0601 05:43:55.420000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01003, (ref-fp64): 0.01001 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4261121Z E0601 05:43:55.425000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00466, (ref-fp64): 0.00467 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4265089Z E0601 05:43:55.426000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00667, (ref-fp64): 0.00669 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4269549Z E0601 05:43:55.426000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01001, (ref-fp64): 0.00998 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4273834Z E0601 05:43:55.426000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00437, (ref-fp64): 0.00437 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4277847Z E0601 05:43:55.427000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00681, (ref-fp64): 0.00679 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4282324Z E0601 05:43:55.427000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01124, (ref-fp64): 0.01122 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4291111Z E0601 05:43:55.428000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00639, (ref-fp64): 0.00638 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4295142Z E0601 05:43:55.429000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00352, (ref-fp64): 0.00347 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4301428Z E0601 05:43:55.429000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00826, (ref-fp64): 0.00822 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4309719Z E0601 05:43:55.430000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00448, (ref-fp64): 0.00447 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4318477Z E0601 05:43:55.431000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00631, (ref-fp64): 0.00629 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4322326Z E0601 05:43:55.431000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01536, (ref-fp64): 0.01512 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4326850Z E0601 05:43:55.432000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01659, (ref-fp64): 0.01642 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4331214Z E0601 05:43:55.432000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.03899, (ref-fp64): 0.03896 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4339479Z E0601 05:43:55.433000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00342, (ref-fp64): 0.00340 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4347900Z E0601 05:43:55.434000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00518, (ref-fp64): 0.00515 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4356455Z E0601 05:43:55.435000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00283, (ref-fp64): 0.00282 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4364987Z E0601 05:43:55.436000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00452, (ref-fp64): 0.00449 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4373382Z E0601 05:43:55.436000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00244, (ref-fp64): 0.00243 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4381685Z E0601 05:43:55.437000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00391, (ref-fp64): 0.00390 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4390305Z E0601 05:43:55.438000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00221, (ref-fp64): 0.00220 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4394179Z E0601 05:43:55.439000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00936, (ref-fp64): 0.00945 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4398541Z E0601 05:43:55.439000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01399, (ref-fp64): 0.01395 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4402786Z E0601 05:43:55.439000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02510, (ref-fp64): 0.02504 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4411455Z E0601 05:43:55.440000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00342, (ref-fp64): 0.00342 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4419837Z E0601 05:43:55.441000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00191, (ref-fp64): 0.00190 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4428258Z E0601 05:43:55.442000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00324, (ref-fp64): 0.00323 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4436627Z E0601 05:43:55.443000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00178, (ref-fp64): 0.00177 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4440538Z E0601 05:43:55.443000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00185, (ref-fp64): 0.00184 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4450069Z E0601 05:43:55.444000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00312, (ref-fp64): 0.00311 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4458447Z E0601 05:43:55.445000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00532, (ref-fp64): 0.00530 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4469709Z E0601 05:43:55.446000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00241, (ref-fp64): 0.00240 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4478295Z E0601 05:43:55.447000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00341, (ref-fp64): 0.00340 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4489666Z E0601 05:43:55.448000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00150, (ref-fp64): 0.00149 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4497962Z E0601 05:43:55.449000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00252, (ref-fp64): 0.00251 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4509299Z E0601 05:43:55.450000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00100, (ref-fp64): 0.00100 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4517718Z E0601 05:43:55.451000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00175, (ref-fp64): 0.00175 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4529025Z E0601 05:43:55.452000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00068, (ref-fp64): 0.00068 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4533012Z E0601 05:43:55.452000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00938, (ref-fp64): 0.00931 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4537128Z E0601 05:43:55.453000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01295, (ref-fp64): 0.01295 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4541899Z E0601 05:43:55.453000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02176, (ref-fp64): 0.02168 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4550600Z E0601 05:43:55.454000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00146, (ref-fp64): 0.00145 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4561855Z E0601 05:43:55.455000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00052, (ref-fp64): 0.00052 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4570344Z E0601 05:43:55.456000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00105, (ref-fp64): 0.00104 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4600952Z E0601 05:43:55.459000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00951, (ref-fp64): 0.00944 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4605139Z E0601 05:43:55.460000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01319, (ref-fp64): 0.01309 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4609559Z E0601 05:43:55.460000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.02338, (ref-fp64): 0.02332 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4687152Z E0601 05:43:55.468000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00587, (ref-fp64): 0.00577 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4691218Z E0601 05:43:55.468000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00806, (ref-fp64): 0.00800 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4695725Z E0601 05:43:55.469000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01392, (ref-fp64): 0.01388 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4766633Z E0601 05:43:55.476000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00037, (ref-fp64): 0.00037 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4774978Z E0601 05:43:55.477000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00088, (ref-fp64): 0.00088 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4828447Z E0601 05:43:55.482000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00617, (ref-fp64): 0.00607 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4832223Z E0601 05:43:55.482000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.00956, (ref-fp64): 0.00942 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4836606Z E0601 05:43:55.483000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01689, (ref-fp64): 0.01684 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4887608Z E0601 05:43:55.488000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01461, (ref-fp64): 0.01366 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4894086Z E0601 05:43:55.489000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01493, (ref-fp64): 0.01486 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4898683Z E0601 05:43:55.489000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01531, (ref-fp64): 0.01536 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4903005Z E0601 05:43:55.489000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01497, (ref-fp64): 0.01480 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4907569Z E0601 05:43:55.490000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4912206Z E0601 05:43:55.490000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01393, (ref-fp64): 0.01422 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4917045Z E0601 05:43:55.491000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01547, (ref-fp64): 0.01536 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4922080Z E0601 05:43:55.491000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01493, (ref-fp64): 0.01493 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4931191Z E0601 05:43:55.492000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01541, (ref-fp64): 0.01548 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4937662Z E0601 05:43:55.493000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01566, (ref-fp64): 0.01566 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4942137Z E0601 05:43:55.493000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01748, (ref-fp64): 0.01745 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4946391Z E0601 05:43:55.494000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01326, (ref-fp64): 0.01329 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4951719Z E0601 05:43:55.494000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01641, (ref-fp64): 0.01640 and shape=torch.Size([128, 384, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4956169Z E0601 05:43:55.495000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01572, (ref-fp64): 0.01573 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4962935Z E0601 05:43:55.495000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01553, (ref-fp64): 0.01553 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4967729Z E0601 05:43:55.496000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01599, (ref-fp64): 0.01610 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4972060Z E0601 05:43:55.496000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01363, (ref-fp64): 0.01369 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4976686Z E0601 05:43:55.497000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01518, (ref-fp64): 0.01518 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4980871Z E0601 05:43:55.497000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01359, (ref-fp64): 0.01360 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4985317Z E0601 05:43:55.498000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01410, (ref-fp64): 0.01412 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.4990330Z E0601 05:43:55.498000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01487, (ref-fp64): 0.01487 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.4999011Z E0601 05:43:55.499000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01386, (ref-fp64): 0.01386 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5008011Z E0601 05:43:55.500000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01340, (ref-fp64): 0.01340 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5016452Z E0601 05:43:55.501000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01460, (ref-fp64): 0.01463 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5020839Z E0601 05:43:55.501000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01500, (ref-fp64): 0.01494 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5025447Z E0601 05:43:55.502000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5030308Z E0601 05:43:55.502000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01558, (ref-fp64): 0.01560 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5035011Z E0601 05:43:55.503000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01462, (ref-fp64): 0.01469 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5039490Z E0601 05:43:55.503000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5044064Z E0601 05:43:55.503000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01424, (ref-fp64): 0.01438 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5048717Z E0601 05:43:55.504000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01408, (ref-fp64): 0.01411 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5053295Z E0601 05:43:55.504000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01480 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5057988Z E0601 05:43:55.505000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01499, (ref-fp64): 0.01479 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5070731Z E0601 05:43:55.505000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01465, (ref-fp64): 0.01454 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5072945Z E0601 05:43:55.506000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01468, (ref-fp64): 0.01466 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5075119Z E0601 05:43:55.506000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01408, (ref-fp64): 0.01406 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5077437Z E0601 05:43:55.507000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01393, (ref-fp64): 0.01400 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5082349Z E0601 05:43:55.507000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01482 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5087112Z E0601 05:43:55.508000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01714, (ref-fp64): 0.01699 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5091485Z E0601 05:43:55.508000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01506, (ref-fp64): 0.01507 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5096119Z E0601 05:43:55.509000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01475, (ref-fp64): 0.01473 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5100480Z E0601 05:43:55.509000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01115, (ref-fp64): 0.01131 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5104833Z E0601 05:43:55.510000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01406, (ref-fp64): 0.01411 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5109451Z E0601 05:43:55.510000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01417, (ref-fp64): 0.01421 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5114052Z E0601 05:43:55.510000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01498, (ref-fp64): 0.01493 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5118574Z E0601 05:43:55.511000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01515, (ref-fp64): 0.01515 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5123339Z E0601 05:43:55.511000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5127897Z E0601 05:43:55.512000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01484 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5132418Z E0601 05:43:55.512000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01650, (ref-fp64): 0.01646 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5137087Z E0601 05:43:55.513000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01477 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5141412Z E0601 05:43:55.513000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01437, (ref-fp64): 0.01432 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5145845Z E0601 05:43:55.514000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01549, (ref-fp64): 0.01558 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5150789Z E0601 05:43:55.514000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01479 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5155413Z E0601 05:43:55.515000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01364, (ref-fp64): 0.01371 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5159873Z E0601 05:43:55.515000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01412, (ref-fp64): 0.01400 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5164562Z E0601 05:43:55.516000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5169209Z E0601 05:43:55.516000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01477, (ref-fp64): 0.01476 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5173593Z E0601 05:43:55.516000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01422, (ref-fp64): 0.01423 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5178522Z E0601 05:43:55.517000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01477, (ref-fp64): 0.01477 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5182964Z E0601 05:43:55.517000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01362, (ref-fp64): 0.01352 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5187424Z E0601 05:43:55.518000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01297, (ref-fp64): 0.01290 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5192447Z E0601 05:43:55.518000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5196606Z E0601 05:43:55.519000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01416, (ref-fp64): 0.01425 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5201307Z E0601 05:43:55.519000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01476 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5206144Z E0601 05:43:55.520000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5210493Z E0601 05:43:55.520000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01625, (ref-fp64): 0.01612 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5214864Z E0601 05:43:55.521000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01615, (ref-fp64): 0.01602 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5219651Z E0601 05:43:55.521000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01467, (ref-fp64): 0.01468 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5224069Z E0601 05:43:55.522000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01491 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5228423Z E0601 05:43:55.522000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01490 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5233326Z E0601 05:43:55.522000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01536, (ref-fp64): 0.01535 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5237661Z E0601 05:43:55.523000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01562, (ref-fp64): 0.01559 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5242401Z E0601 05:43:55.523000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01620, (ref-fp64): 0.01610 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5247481Z E0601 05:43:55.524000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01484 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5251548Z E0601 05:43:55.524000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01501, (ref-fp64): 0.01521 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5255923Z E0601 05:43:55.525000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01425, (ref-fp64): 0.01404 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5260785Z E0601 05:43:55.525000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01566, (ref-fp64): 0.01566 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5264870Z E0601 05:43:55.526000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01399, (ref-fp64): 0.01401 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5269333Z E0601 05:43:55.526000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01463, (ref-fp64): 0.01463 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5274363Z E0601 05:43:55.527000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01481 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5278785Z E0601 05:43:55.527000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01532, (ref-fp64): 0.01535 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5283292Z E0601 05:43:55.527000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01472, (ref-fp64): 0.01476 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5290722Z E0601 05:43:55.528000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01484, (ref-fp64): 0.01484 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5294772Z E0601 05:43:55.529000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01440, (ref-fp64): 0.01447 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5299427Z E0601 05:43:55.529000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01471, (ref-fp64): 0.01479 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5304181Z E0601 05:43:55.530000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5308524Z E0601 05:43:55.530000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01505, (ref-fp64): 0.01518 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5312984Z E0601 05:43:55.530000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01510, (ref-fp64): 0.01511 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5320271Z E0601 05:43:55.531000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5324899Z E0601 05:43:55.532000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01590, (ref-fp64): 0.01594 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5330164Z E0601 05:43:55.532000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01382, (ref-fp64): 0.01386 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5335473Z E0601 05:43:55.533000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01466, (ref-fp64): 0.01466 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5340374Z E0601 05:43:55.533000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01455, (ref-fp64): 0.01447 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5345682Z E0601 05:43:55.534000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01548, (ref-fp64): 0.01543 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5353034Z E0601 05:43:55.534000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5357674Z E0601 05:43:55.535000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01578, (ref-fp64): 0.01565 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5362674Z E0601 05:43:55.535000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01389 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5367711Z E0601 05:43:55.536000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01475, (ref-fp64): 0.01475 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5371947Z E0601 05:43:55.536000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01476 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5376675Z E0601 05:43:55.537000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01446, (ref-fp64): 0.01448 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5383988Z E0601 05:43:55.538000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5388402Z E0601 05:43:55.538000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01538, (ref-fp64): 0.01538 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5392751Z E0601 05:43:55.538000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01414, (ref-fp64): 0.01418 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5397687Z E0601 05:43:55.539000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5402043Z E0601 05:43:55.539000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01482, (ref-fp64): 0.01472 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5406754Z E0601 05:43:55.540000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01593, (ref-fp64): 0.01596 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5414357Z E0601 05:43:55.541000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5418670Z E0601 05:43:55.541000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01484, (ref-fp64): 0.01483 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5422893Z E0601 05:43:55.541000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01504, (ref-fp64): 0.01513 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5427878Z E0601 05:43:55.542000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5432601Z E0601 05:43:55.542000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01523, (ref-fp64): 0.01537 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5436864Z E0601 05:43:55.543000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01347, (ref-fp64): 0.01347 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5442514Z E0601 05:43:55.543000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01470, (ref-fp64): 0.01470 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5446475Z E0601 05:43:55.544000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01532, (ref-fp64): 0.01530 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5451015Z E0601 05:43:55.544000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01449, (ref-fp64): 0.01449 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5458264Z E0601 05:43:55.545000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01481 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5464575Z E0601 05:43:55.545000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01447, (ref-fp64): 0.01466 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5468614Z E0601 05:43:55.546000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01380, (ref-fp64): 0.01376 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5473791Z E0601 05:43:55.546000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01489, (ref-fp64): 0.01489 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5478028Z E0601 05:43:55.547000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01450, (ref-fp64): 0.01438 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5482411Z E0601 05:43:55.547000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01511, (ref-fp64): 0.01508 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5489731Z E0601 05:43:55.548000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5494156Z E0601 05:43:55.549000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01592, (ref-fp64): 0.01610 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5498643Z E0601 05:43:55.549000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01395, (ref-fp64): 0.01398 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5503325Z E0601 05:43:55.549000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01527, (ref-fp64): 0.01527 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5507716Z E0601 05:43:55.550000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01502, (ref-fp64): 0.01497 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5512488Z E0601 05:43:55.550000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01461, (ref-fp64): 0.01460 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5519567Z E0601 05:43:55.551000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5523959Z E0601 05:43:55.552000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01479, (ref-fp64): 0.01474 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5528545Z E0601 05:43:55.552000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01493, (ref-fp64): 0.01486 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5533408Z E0601 05:43:55.552000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01596, (ref-fp64): 0.01597 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5538668Z E0601 05:43:55.553000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01571, (ref-fp64): 0.01566 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5542824Z E0601 05:43:55.553000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01612, (ref-fp64): 0.01604 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5547361Z E0601 05:43:55.554000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01482, (ref-fp64): 0.01484 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5551926Z E0601 05:43:55.554000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01484, (ref-fp64): 0.01484 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5556508Z E0601 05:43:55.555000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01473, (ref-fp64): 0.01475 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5563753Z E0601 05:43:55.555000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5568815Z E0601 05:43:55.556000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01437, (ref-fp64): 0.01430 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5572961Z E0601 05:43:55.556000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01442, (ref-fp64): 0.01444 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5595567Z E0601 05:43:55.559000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5599771Z E0601 05:43:55.559000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01398, (ref-fp64): 0.01395 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5604332Z E0601 05:43:55.560000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01441, (ref-fp64): 0.01446 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5609546Z E0601 05:43:55.560000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01462, (ref-fp64): 0.01462 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5613801Z E0601 05:43:55.561000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01481, (ref-fp64): 0.01481 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5618323Z E0601 05:43:55.561000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01401, (ref-fp64): 0.01399 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5641508Z E0601 05:43:55.563000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01467, (ref-fp64): 0.01467 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5645929Z E0601 05:43:55.564000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01511, (ref-fp64): 0.01517 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5650258Z E0601 05:43:55.564000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01506, (ref-fp64): 0.01503 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5655293Z E0601 05:43:55.565000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01461, (ref-fp64): 0.01461 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5659849Z E0601 05:43:55.565000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01459, (ref-fp64): 0.01457 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5664144Z E0601 05:43:55.566000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01512, (ref-fp64): 0.01513 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5686958Z E0601 05:43:55.568000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01463, (ref-fp64): 0.01463 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5690971Z E0601 05:43:55.568000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01490, (ref-fp64): 0.01476 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5695859Z E0601 05:43:55.569000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01456, (ref-fp64): 0.01453 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5700659Z E0601 05:43:55.569000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01452 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5705114Z E0601 05:43:55.570000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01425, (ref-fp64): 0.01450 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5709416Z E0601 05:43:55.570000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01452, (ref-fp64): 0.01444 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5715053Z E0601 05:43:55.571000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01487, (ref-fp64): 0.01486 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5719257Z E0601 05:43:55.571000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01417, (ref-fp64): 0.01418 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5723878Z E0601 05:43:55.571000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01431, (ref-fp64): 0.01420 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5746707Z E0601 05:43:55.574000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01455, (ref-fp64): 0.01455 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5751057Z E0601 05:43:55.574000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01449, (ref-fp64): 0.01454 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5755480Z E0601 05:43:55.575000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01471, (ref-fp64): 0.01473 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5760258Z E0601 05:43:55.575000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01448, (ref-fp64): 0.01448 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5764828Z E0601 05:43:55.576000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01429, (ref-fp64): 0.01420 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5769409Z E0601 05:43:55.576000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01442, (ref-fp64): 0.01443 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5792280Z E0601 05:43:55.578000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01450, (ref-fp64): 0.01450 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5796670Z E0601 05:43:55.579000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01348, (ref-fp64): 0.01360 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5800925Z E0601 05:43:55.579000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01389, (ref-fp64): 0.01393 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5806035Z E0601 05:43:55.580000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01426, (ref-fp64): 0.01427 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5810551Z E0601 05:43:55.580000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01395, (ref-fp64): 0.01388 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5814847Z E0601 05:43:55.581000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01443, (ref-fp64): 0.01443 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5837613Z E0601 05:43:55.583000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01428, (ref-fp64): 0.01428 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5841799Z E0601 05:43:55.583000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01392, (ref-fp64): 0.01389 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5846704Z E0601 05:43:55.584000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01472, (ref-fp64): 0.01474 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5851579Z E0601 05:43:55.584000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01412, (ref-fp64): 0.01412 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5855824Z E0601 05:43:55.585000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01253, (ref-fp64): 0.01252 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5860184Z E0601 05:43:55.585000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01280, (ref-fp64): 0.01284 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5867075Z E0601 05:43:55.586000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01405, (ref-fp64): 0.01405 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5871531Z E0601 05:43:55.586000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01362, (ref-fp64): 0.01364 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5876204Z E0601 05:43:55.587000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01280, (ref-fp64): 0.01280 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5898847Z E0601 05:43:55.589000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01359, (ref-fp64): 0.01359 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5903242Z E0601 05:43:55.589000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01280, (ref-fp64): 0.01278 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5907594Z E0601 05:43:55.590000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01249, (ref-fp64): 0.01248 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5912587Z E0601 05:43:55.590000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01321, (ref-fp64): 0.01321 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5939309Z E0601 05:43:55.593000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01293, (ref-fp64): 0.01293 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.5948010Z E0601 05:43:55.594000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01338, (ref-fp64): 0.01343 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5952553Z E0601 05:43:55.594000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01506, (ref-fp64): 0.01493 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5957308Z E0601 05:43:55.595000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01475, (ref-fp64): 0.01476 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5961410Z E0601 05:43:55.595000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01538, (ref-fp64): 0.01551 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5966301Z E0601 05:43:55.596000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01460, (ref-fp64): 0.01461 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5970983Z E0601 05:43:55.596000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01445, (ref-fp64): 0.01445 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5975201Z E0601 05:43:55.597000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01495, (ref-fp64): 0.01493 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5979718Z E0601 05:43:55.597000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01402, (ref-fp64): 0.01401 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5984445Z E0601 05:43:55.598000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([256, 768, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5988809Z E0601 05:43:55.598000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01377, (ref-fp64): 0.01379 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.5993530Z E0601 05:43:55.598000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01401, (ref-fp64): 0.01402 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6000700Z E0601 05:43:55.599000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01447, (ref-fp64): 0.01447 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.6005280Z E0601 05:43:55.600000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01336, (ref-fp64): 0.01345 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6009560Z E0601 05:43:55.600000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01327, (ref-fp64): 0.01328 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6014499Z E0601 05:43:55.601000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01414, (ref-fp64): 0.01415 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6019098Z E0601 05:43:55.601000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01242, (ref-fp64): 0.01240 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6023367Z E0601 05:43:55.601000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01267, (ref-fp64): 0.01264 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6030827Z E0601 05:43:55.602000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01421, (ref-fp64): 0.01421 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.6035481Z E0601 05:43:55.603000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01177, (ref-fp64): 0.01180 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6040065Z E0601 05:43:55.603000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01307, (ref-fp64): 0.01308 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6044959Z E0601 05:43:55.604000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01349, (ref-fp64): 0.01349 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-01T05:43:55.6056202Z E0601 05:43:55.605000 139828816900736 torch/_dynamo/utils.py:1401] RMSE (res-fp64): 0.01264, (ref-fp64): 0.01264 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000 2024-06-01T05:43:55.6530250Z pass 2024-06-01T05:43:55.7001070Z TIMING: entire_frame_compile:81.57003 inductor_compile:41.72536 backend_compile:57.63652 code_gen:28.07782 2024-06-01T05:43:55.7002830Z STATS: call_* op count: 743 | FakeTensor.__torch_dispatch__:13760 | FakeTensorMode.__torch_dispatch__:86411 | attempt fast:1294 | fast is_contiguous:1294 | ProxyTorchDispatchMode.__torch_dispatch__:10810 2024-06-01T05:43:55.7004974Z Dynamo produced 31 graphs covering 743 ops with 9 graph breaks (6 unique) 2024-06-01T05:43:59.8843784Z accuracy pass_rate=76.32% 2024-06-01T05:43:59.8846164Z calls_captured gmean=0.00x mean=399.079x 2024-06-01T05:43:59.8850064Z unique_graphs gmean=0.00x mean=4.342x 2024-06-01T05:43:59.8853466Z graph_breaks gmean=0.00x mean=6.211x 2024-06-01T05:43:59.8856767Z unique_graph_breaks gmean=0.00x mean=4.289x 2024-06-01T05:43:59.8860112Z autograd_captures gmean=0.00x mean=0.000x 2024-06-01T05:43:59.8863297Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-01T05:43:59.8866668Z cudagraph_skips gmean=0.00x mean=0.737x 2024-06-01T05:44:00.9658186Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/dynamic_inductor_torchbench_training.csv 2024-06-01T05:44:01.2795959Z lennard_jones PASS 2024-06-01T05:44:01.2799164Z llava XFAIL 2024-06-01T05:44:01.2804753Z maml_omniglot PASS 2024-06-01T05:44:01.2809006Z mnasnet1_0 PASS 2024-06-01T05:44:01.2813390Z mobilenet_v2 PASS 2024-06-01T05:44:01.2817723Z mobilenet_v2_quantized_qat XFAIL 2024-06-01T05:44:01.2822210Z mobilenet_v3_large PASS 2024-06-01T05:44:01.2826617Z moco PASS 2024-06-01T05:44:01.2831113Z nanogpt PASS 2024-06-01T05:44:01.2835762Z nvidia_deeprecommender PASS 2024-06-01T05:44:01.2839896Z opacus_cifar10 XFAIL 2024-06-01T05:44:01.2844494Z phlippe_densenet PASS 2024-06-01T05:44:01.2848967Z phlippe_resnet FAIL: accuracy=fail_accuracy, expected=pass 2024-06-01T05:44:01.2853296Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-01T05:44:01.2857666Z pytorch_stargan PASS 2024-06-01T05:44:01.2862176Z pytorch_unet XFAIL 2024-06-01T05:44:01.2866696Z resnet152 PASS 2024-06-01T05:44:01.2871256Z resnet18 PASS 2024-06-01T05:44:01.2875708Z resnet50 PASS 2024-06-01T05:44:01.2880047Z resnet50_quantized_qat XFAIL 2024-06-01T05:44:01.2884741Z resnext50_32x4d PASS 2024-06-01T05:44:01.2889071Z sam XFAIL 2024-06-01T05:44:01.2893539Z shufflenet_v2_x1_0 PASS 2024-06-01T05:44:01.2897912Z soft_actor_critic PASS 2024-06-01T05:44:01.2902358Z squeezenet1_1 PASS 2024-06-01T05:44:01.2906653Z stable_diffusion_text_encoder PASS 2024-06-01T05:44:01.2911175Z stable_diffusion_unet XFAIL 2024-06-01T05:44:01.2915609Z timm_efficientnet PASS 2024-06-01T05:44:01.2919895Z timm_regnet PASS 2024-06-01T05:44:01.2924569Z timm_resnest PASS 2024-06-01T05:44:01.2929020Z timm_vision_transformer PASS 2024-06-01T05:44:01.2933630Z timm_vision_transformer_large XFAIL 2024-06-01T05:44:01.2937906Z timm_vovnet PASS 2024-06-01T05:44:01.2942810Z torch_multimodal_clip PASS 2024-06-01T05:44:01.2947151Z tts_angular PASS 2024-06-01T05:44:01.2951866Z vgg16 PASS 2024-06-01T05:44:01.2956106Z vision_maskrcnn PASS 2024-06-01T05:44:01.2960720Z yolov3 PASS 2024-06-01T05:44:01.2961030Z 2024-06-01T05:44:01.2961226Z Error: 1 models have accuracy status regressed: 2024-06-01T05:44:01.2961706Z phlippe_resnet 2024-06-01T05:44:01.2961916Z 2024-06-01T05:44:01.2964385Z 2024-06-01T05:44:01.2966547Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/cu124/dynamic_inductor_torchbench_training.csv` to reflect the new baseline. 2024-06-01T05:44:01.2967677Z from pytorch/pytorch root, run 2024-06-01T05:44:01.2968468Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py de352ff31081bc3b80baf4f72168a00bdf6cccae` 2024-06-01T05:44:01.2969470Z and then `git add` the resulting local changes to expected CSVs to your commit. 2024-06-01T05:44:01.2969986Z 2024-06-01T05:44:01.3418664Z + cleanup_workspace 2024-06-01T05:44:01.3419586Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-01T05:44:01.3420730Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-01T05:44:01.3421684Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-01T05:44:01.3422345Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-01T05:44:01.3423169Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-01T05:44:01.3424074Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-01T05:44:01.3424814Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-01T05:44:01.8683289Z ##[error]Process completed with exit code 1. 2024-06-01T05:44:01.8757892Z Prepare all required actions 2024-06-01T05:44:01.8758332Z Getting action download info 2024-06-01T05:44:02.0224700Z ##[group]Run ./.github/actions/pytest-cache-upload 2024-06-01T05:44:02.0225189Z with: 2024-06-01T05:44:02.0225470Z cache_dir: .pytest_cache 2024-06-01T05:44:02.0225823Z shard: 2 2024-06-01T05:44:02.0226172Z sha: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T05:44:02.0226672Z test_config: dynamic_inductor_torchbench 2024-06-01T05:44:02.0227252Z job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T05:44:02.0227786Z env: 2024-06-01T05:44:02.0228066Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:02.0228529Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:02.0229275Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:02.0230283Z ##[endgroup] 2024-06-01T05:44:02.0269398Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-01T05:44:02.0270174Z with: 2024-06-01T05:44:02.0270450Z shell: bash 2024-06-01T05:44:02.0270746Z timeout_minutes: 5 2024-06-01T05:44:02.0271073Z max_attempts: 5 2024-06-01T05:44:02.0271394Z retry_wait_seconds: 30 2024-06-01T05:44:02.0271841Z command: set -eu python3 -m pip install boto3==1.19.12 2024-06-01T05:44:02.0272365Z polling_interval_seconds: 1 2024-06-01T05:44:02.0272741Z warning_on_retry: true 2024-06-01T05:44:02.0273096Z continue_on_error: false 2024-06-01T05:44:02.0273430Z env: 2024-06-01T05:44:02.0273707Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:02.0274164Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:02.0274911Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:02.0275550Z ##[endgroup] 2024-06-01T05:44:02.2938596Z Defaulting to user installation because normal site-packages is not writeable 2024-06-01T05:44:03.5139197Z Collecting boto3==1.19.12 2024-06-01T05:44:03.5317397Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2024-06-01T05:44:03.5936576Z Collecting s3transfer<0.6.0,>=0.5.0 2024-06-01T05:44:03.5974008Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-06-01T05:44:03.6339721Z Collecting jmespath<1.0.0,>=0.7.1 2024-06-01T05:44:03.6375873Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-06-01T05:44:05.0684466Z Collecting botocore<1.23.0,>=1.22.12 2024-06-01T05:44:05.0759361Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2024-06-01T05:44:05.2311494Z Collecting python-dateutil<3.0.0,>=2.1 2024-06-01T05:44:05.2348161Z Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-06-01T05:44:05.2466512Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2024-06-01T05:44:05.2992569Z Collecting six>=1.5 2024-06-01T05:44:05.3035199Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-06-01T05:44:05.3740368Z Installing collected packages: six, python-dateutil, jmespath, botocore, s3transfer, boto3 2024-06-01T05:44:06.0396981Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0 2024-06-01T05:44:07.0813978Z Command completed after 1 attempt(s). 2024-06-01T05:44:07.0870881Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2024-06-01T05:44:07.0871493Z python3 .github/scripts/pytest_cache.py \ 2024-06-01T05:44:07.0871983Z  --upload \ 2024-06-01T05:44:07.0872407Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2024-06-01T05:44:07.0872921Z  --pr_identifier $GITHUB_REF \ 2024-06-01T05:44:07.0873415Z  --job_identifier $JOB_IDENTIFIER \ 2024-06-01T05:44:07.0873878Z  --sha $SHA \ 2024-06-01T05:44:07.0874257Z  --test_config $TEST_CONFIG \ 2024-06-01T05:44:07.0874694Z  --shard $SHARD \ 2024-06-01T05:44:07.0875069Z  --repo $REPO \ 2024-06-01T05:44:07.0875558Z  --temp_dir $RUNNER_TEMP \ 2024-06-01T05:44:07.0883288Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.0883809Z env: 2024-06-01T05:44:07.0884087Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.0884549Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.0885291Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.0885957Z CACHE_DIR: .pytest_cache 2024-06-01T05:44:07.0886456Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-06-01T05:44:07.0887060Z SHA: de352ff31081bc3b80baf4f72168a00bdf6cccae 2024-06-01T05:44:07.0887553Z TEST_CONFIG: dynamic_inductor_torchbench 2024-06-01T05:44:07.0887980Z SHARD: 2 2024-06-01T05:44:07.0888268Z REPO: pytorch/pytorch 2024-06-01T05:44:07.0888605Z ##[endgroup] 2024-06-01T05:44:07.2220131Z PR identifier for `refs/tags/ciflow/inductor/127669` is `b0b5e6abbfe2c3236a81c6be5b165415` 2024-06-01T05:44:07.2223566Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/127669', repo='pytorch/pytorch', sha='de352ff31081bc3b80baf4f72168a00bdf6cccae', shard='2', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='dynamic_inductor_torchbench', upload=True) 2024-06-01T05:44:07.2226616Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2024-06-01T05:44:07.2388665Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-01T05:44:07.2389191Z cat test/**/*_toprint.log || true 2024-06-01T05:44:07.2397088Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.2397610Z env: 2024-06-01T05:44:07.2397894Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.2398371Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.2399125Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.2399781Z ##[endgroup] 2024-06-01T05:44:07.2464375Z cat: test/**/*_toprint.log: No such file or directory 2024-06-01T05:44:07.2492780Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-01T05:44:07.2493264Z kill "$MONITOR_SCRIPT_PID" 2024-06-01T05:44:07.2499787Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.2500294Z env: 2024-06-01T05:44:07.2500601Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.2501091Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.2501858Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.2502528Z MONITOR_SCRIPT_PID: 4938 2024-06-01T05:44:07.2502878Z ##[endgroup] 2024-06-01T05:44:07.2695559Z Prepare all required actions 2024-06-01T05:44:07.2696018Z Getting action download info 2024-06-01T05:44:07.3642990Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-01T05:44:07.5065283Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-01T05:44:07.5065782Z with: 2024-06-01T05:44:07.5066375Z file-suffix: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171 2024-06-01T05:44:07.5067116Z s3-bucket: gha-artifacts 2024-06-01T05:44:07.5067462Z env: 2024-06-01T05:44:07.5067750Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5068217Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5068978Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5069636Z ##[endgroup] 2024-06-01T05:44:07.5094824Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-01T05:44:07.5095637Z # Remove any previous test jsons if they exist 2024-06-01T05:44:07.5096185Z rm -f test-jsons-*.zip 2024-06-01T05:44:07.5096721Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-01T05:44:07.5104185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.5104702Z env: 2024-06-01T05:44:07.5104990Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5105451Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5106201Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5107162Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171 2024-06-01T05:44:07.5107866Z ##[endgroup] 2024-06-01T05:44:07.5305154Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-01T05:44:07.5336290Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-01T05:44:07.5337008Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-01T05:44:07.5341236Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-01T05:44:07.5385128Z ##[group]Run # Remove any previous test reports if they exist 2024-06-01T05:44:07.5385783Z # Remove any previous test reports if they exist 2024-06-01T05:44:07.5386315Z rm -f test-reports-*.zip 2024-06-01T05:44:07.5386926Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-01T05:44:07.5394664Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.5395174Z env: 2024-06-01T05:44:07.5395454Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5395917Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5396663Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5397607Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171 2024-06-01T05:44:07.5398299Z ##[endgroup] 2024-06-01T05:44:07.5556975Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-06-01T05:44:07.5559522Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-06-01T05:44:07.5560926Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 80%) 2024-06-01T05:44:07.5562233Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-06-01T05:44:07.5568901Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-01T05:44:07.5570441Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-01T05:44:07.5596281Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-01T05:44:07.5596929Z # Remove any previous usage logs if they exist 2024-06-01T05:44:07.5597442Z rm -f logs-*.zip 2024-06-01T05:44:07.5598132Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-01T05:44:07.5599299Z # so check to see if the file exists first 2024-06-01T05:44:07.5599816Z if [ -f 'usage_log.txt' ]; then 2024-06-01T05:44:07.5600356Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-01T05:44:07.5600855Z fi 2024-06-01T05:44:07.5601229Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-01T05:44:07.5601804Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-01T05:44:07.5602402Z fi 2024-06-01T05:44:07.5609447Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.5609962Z env: 2024-06-01T05:44:07.5610241Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5610703Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5611462Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5612417Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171 2024-06-01T05:44:07.5613116Z ##[endgroup] 2024-06-01T05:44:07.5704967Z adding: usage_log.txt (deflated 92%) 2024-06-01T05:44:07.5770856Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-01T05:44:07.5771606Z # Remove any previous debugging artifacts if they exist 2024-06-01T05:44:07.5772176Z rm -f debug-*.zip 2024-06-01T05:44:07.5772574Z if [ -d 'test/debug' ]; then 2024-06-01T05:44:07.5773089Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-01T05:44:07.5773580Z fi 2024-06-01T05:44:07.5780787Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:07.5781298Z env: 2024-06-01T05:44:07.5781572Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5782036Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5782791Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5783758Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171 2024-06-01T05:44:07.5784477Z ##[endgroup] 2024-06-01T05:44:07.5858519Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-01T05:44:07.5859091Z with: 2024-06-01T05:44:07.5859377Z s3-bucket: gha-artifacts 2024-06-01T05:44:07.5859815Z s3-prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:07.5860300Z retention-days: 14 2024-06-01T05:44:07.5860637Z if-no-files-found: warn 2024-06-01T05:44:07.5861000Z path: test-jsons-*.zip 2024-06-01T05:44:07.5861346Z name: artifact 2024-06-01T05:44:07.5861651Z region: us-east-1 2024-06-01T05:44:07.5861948Z env: 2024-06-01T05:44:07.5862222Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:07.5862684Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:07.5863428Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:07.5864076Z ##[endgroup] 2024-06-01T05:44:07.8944301Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-01T05:44:07.8945005Z With the provided path, there will be 1 file uploaded 2024-06-01T05:44:07.8945695Z Uploading to s3 prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:08.0102356Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:08.1449803Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:08.1593101Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-01T05:44:08.1593550Z with: 2024-06-01T05:44:08.1593838Z s3-bucket: gha-artifacts 2024-06-01T05:44:08.1594278Z s3-prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:08.1594764Z retention-days: 14 2024-06-01T05:44:08.1595105Z if-no-files-found: error 2024-06-01T05:44:08.1595480Z path: test-reports-*.zip 2024-06-01T05:44:08.1595836Z name: artifact 2024-06-01T05:44:08.1596137Z region: us-east-1 2024-06-01T05:44:08.1596440Z env: 2024-06-01T05:44:08.1596717Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:08.1597292Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:08.1598046Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:08.1598709Z ##[endgroup] 2024-06-01T05:44:08.4677946Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-01T05:44:08.4678598Z With the provided path, there will be 1 file uploaded 2024-06-01T05:44:08.4679250Z Uploading to s3 prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:08.4704652Z Starting upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:08.6572688Z Finished upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:08.7026447Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-01T05:44:08.7026901Z with: 2024-06-01T05:44:08.7027194Z s3-bucket: gha-artifacts 2024-06-01T05:44:08.7027626Z s3-prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:08.7028101Z retention-days: 14 2024-06-01T05:44:08.7028452Z if-no-files-found: ignore 2024-06-01T05:44:08.7028827Z path: logs-*.zip 2024-06-01T05:44:08.7029137Z name: artifact 2024-06-01T05:44:08.7029434Z region: us-east-1 2024-06-01T05:44:08.7029733Z env: 2024-06-01T05:44:08.7030272Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:08.7030744Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:08.7031490Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:08.7032137Z ##[endgroup] 2024-06-01T05:44:09.0171847Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-01T05:44:09.0172477Z With the provided path, there will be 1 file uploaded 2024-06-01T05:44:09.0173121Z Uploading to s3 prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:09.0198767Z Starting upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:09.1573440Z Finished upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25675761171.zip 2024-06-01T05:44:09.1713872Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-01T05:44:09.1714439Z with: 2024-06-01T05:44:09.1714725Z s3-bucket: gha-artifacts 2024-06-01T05:44:09.1715161Z s3-prefix: pytorch/pytorch/9326485603/1/artifact 2024-06-01T05:44:09.1715645Z retention-days: 14 2024-06-01T05:44:09.1715984Z if-no-files-found: ignore 2024-06-01T05:44:09.1716351Z path: debug-*.zip 2024-06-01T05:44:09.1716669Z name: artifact 2024-06-01T05:44:09.1716970Z region: us-east-1 2024-06-01T05:44:09.1717275Z env: 2024-06-01T05:44:09.1717556Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:09.1718022Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:09.1718773Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:09.1719429Z ##[endgroup] 2024-06-01T05:44:09.4757204Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-01T05:44:09.4927012Z ##[group]Run # shellcheck disable=SC2156 2024-06-01T05:44:09.4927502Z # shellcheck disable=SC2156 2024-06-01T05:44:09.4928332Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-01T05:44:09.4935969Z shell: /usr/bin/bash -e {0} 2024-06-01T05:44:09.4936318Z env: 2024-06-01T05:44:09.4936601Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:09.4937061Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:09.4937810Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:09.4938450Z ##[endgroup] 2024-06-01T05:44:09.6630988Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-01T05:44:09.6631456Z with: 2024-06-01T05:44:09.6631970Z name: coredumps-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu 2024-06-01T05:44:09.6632626Z retention-days: 14 2024-06-01T05:44:09.6632966Z if-no-files-found: ignore 2024-06-01T05:44:09.6633335Z path: ./**/core.[1-9]* 2024-06-01T05:44:09.6633693Z s3-bucket: gha-artifacts 2024-06-01T05:44:09.6634061Z region: us-east-1 2024-06-01T05:44:09.6634353Z env: 2024-06-01T05:44:09.6634629Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:09.6635086Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:09.6635841Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:09.6636488Z ##[endgroup] 2024-06-01T05:44:19.0610306Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2024-06-01T05:44:19.0837986Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-01T05:44:19.0838571Z with: 2024-06-01T05:44:19.0838825Z env: 2024-06-01T05:44:19.0839100Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:19.0839553Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:19.0840309Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:19.0840966Z ##[endgroup] 2024-06-01T05:44:19.0859964Z ##[group]Run set -eou pipefail 2024-06-01T05:44:19.0860388Z set -eou pipefail 2024-06-01T05:44:19.0860742Z  2024-06-01T05:44:19.0861267Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-01T05:44:19.0861915Z for _ in $(seq 1440); do 2024-06-01T05:44:19.0862400Z  # Break if no ssh session exists anymore 2024-06-01T05:44:19.0862904Z  if [ "$(who)" = "" ]; then 2024-06-01T05:44:19.0863320Z  break 2024-06-01T05:44:19.0863659Z  fi 2024-06-01T05:44:19.0863958Z  echo "." 2024-06-01T05:44:19.0864288Z  sleep 5 2024-06-01T05:44:19.0864606Z done 2024-06-01T05:44:19.0872180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:19.0872687Z env: 2024-06-01T05:44:19.0872962Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:19.0873424Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:19.0874224Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:19.0874971Z ##[endgroup] 2024-06-01T05:44:19.0894763Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-01T05:44:19.0956260Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-01T05:44:19.0957047Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-01T05:44:19.0957656Z # shellcheck disable=SC2046 2024-06-01T05:44:19.0958138Z docker stop $(docker ps -q) || true 2024-06-01T05:44:19.0958627Z # Prune all of the docker images 2024-06-01T05:44:19.0959095Z docker system prune -af 2024-06-01T05:44:19.0966027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:19.0966536Z env: 2024-06-01T05:44:19.0966804Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:19.0967259Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:19.0968010Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:19.0968663Z ##[endgroup] 2024-06-01T05:44:19.6736252Z 115e64e466b1 2024-06-01T05:44:22.0320158Z Deleted Containers: 2024-06-01T05:44:22.0320862Z 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:22.0321414Z 2024-06-01T05:44:25.9761723Z Deleted Images: 2024-06-01T05:44:25.9763753Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:7790448f81f0f3396d69a76eba86a4be7ac35343 2024-06-01T05:44:25.9766291Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks@sha256:b8296027e5f1cfef6dbf6c011e0b1f6ebfa606876a8eb59b07c03b8046ec8490 2024-06-01T05:44:25.9767892Z deleted: sha256:d37233509623aba91eac13ad223e1b9eed85dd9651dcb816d1c4217e406e176c 2024-06-01T05:44:25.9768770Z deleted: sha256:9e2154ff01319a8da3104c4ae4dbf0b9f08dc2f6f8899bb6c494d42dae468313 2024-06-01T05:44:25.9769645Z deleted: sha256:160bf468fc15dd9c533bb8a6df8bcf37607621ba672c6a23361ed52d1bd899b9 2024-06-01T05:44:25.9770513Z deleted: sha256:cf72e9442353c9a16364140452dc60752ee360d5e40f40f721fb195fe145802a 2024-06-01T05:44:25.9771353Z deleted: sha256:6d6e52a9fe43546e6ce46f080d8463e756bfc49c4b4c8c161c1502642ef95ec6 2024-06-01T05:44:25.9772222Z deleted: sha256:b2a8cdfbe8864433266101f6c641c10a9d0fd1031d7b0d10a09c7f4174a5fb5f 2024-06-01T05:44:25.9773110Z deleted: sha256:de356dfb59e8972e92cb24afc65cc46f3ccc1be364ffd10f8d637d25531b938a 2024-06-01T05:44:25.9774174Z deleted: sha256:6330cccf0524696dcea14dbe8b39ef3f0a9871b77af2985d15e8d7d6192dbf0d 2024-06-01T05:44:25.9775085Z deleted: sha256:2681c723013b5eb9e31df7c246b987cdeda079137f4475a0385441576f6fdc9f 2024-06-01T05:44:25.9775935Z deleted: sha256:529969f7d6a71511fd0fe1f35c81c4f55549b40de99907c4f89f77229284c5a3 2024-06-01T05:44:25.9776799Z deleted: sha256:3494b2faa295fc1385286d895b8b0e84b612fea4941beae53f49066c5e63b562 2024-06-01T05:44:25.9777680Z deleted: sha256:f98f6002c5a9cbbaacf0baf36f1cd31fe83fcacb420bc046911610fbb0953b2e 2024-06-01T05:44:25.9778568Z deleted: sha256:cb244980f716a49f312362b063e336d0ccc8358fc15a4abcfc5b55e4e579cbba 2024-06-01T05:44:25.9779509Z deleted: sha256:96d3bb95fcf16282c75c59fc546b442621621cae2601cf127d1d14ad689e676a 2024-06-01T05:44:25.9780574Z deleted: sha256:f1e1c76a9efb3d43fa8564441a460a1ed1c8aeec40d978b13d68a79c001e2804 2024-06-01T05:44:25.9781448Z deleted: sha256:226ffc6ce7c2382accedd80b609e0e294f3b00d4d16953b41a20ebab58384fdf 2024-06-01T05:44:25.9782324Z deleted: sha256:6bddb678ad10cc4fedd10b04b8879cf298ec92b0cf4a728a650d9d19e20a7a9d 2024-06-01T05:44:25.9783197Z deleted: sha256:bcda853175431be68b2e0cb3a2daf273af530abaa43227621293152e7adf2df7 2024-06-01T05:44:25.9784056Z deleted: sha256:dd052c860bf02e8ec5142bcd0b22590e63aa73928cc458985b64bfcdc62cb373 2024-06-01T05:44:25.9784908Z deleted: sha256:14c145900169351e379febaf6fa73cd6c57f6bfb35976e6adc0750a9f1d57641 2024-06-01T05:44:25.9785755Z deleted: sha256:b4b0726096efc52ad832a69c0e14fd193c419d1813448fb042c76a5b7361a827 2024-06-01T05:44:25.9786812Z deleted: sha256:5a5cea9e7f41be015d02cef0257fe7c0790066ec890be1a188eece2aa988f2bd 2024-06-01T05:44:25.9787680Z deleted: sha256:c211c4a4ae3d572450aaaeb5c2fe84e50a761a0da7ad7a08a0bb74017690641c 2024-06-01T05:44:25.9788541Z deleted: sha256:fa72c90d812ac8d2f2ada3e2238668f680e48645deaf32abaa2521640e8ddb56 2024-06-01T05:44:25.9789389Z deleted: sha256:98f1f95b5cd253ae559de0d6468d222a222b4878c9b296b6eaf9161451a0dca8 2024-06-01T05:44:25.9790528Z deleted: sha256:0ffe2cd135db2755026cdd9021072abd82ecc45192965d9b1b4cfd5e17069967 2024-06-01T05:44:25.9791404Z deleted: sha256:6ec4e39869d4166bcfcdf3eb40ab40664346010a9ae8c34f8d1a7c1af7fded12 2024-06-01T05:44:25.9792255Z deleted: sha256:596ef863f6845f2932ca3319315b9c21add2947145f9176f9012ba8d878b9bf6 2024-06-01T05:44:25.9793094Z deleted: sha256:d61bff1e7381058c9c7f2332dffe770a949fb5f0ab716543177262921d70ed10 2024-06-01T05:44:25.9793938Z deleted: sha256:f3c19a03bd75a0259770a3783ab46182c56ccad7dd2461383b94a30e491049e6 2024-06-01T05:44:25.9794783Z deleted: sha256:bd5288b7c649362bcea62570c61b965063af86b7b6684a41655a74a4cf6d20c0 2024-06-01T05:44:25.9795650Z deleted: sha256:8629113d903e6a3e0b4e6c30e3eb34257facb26aeb2acddfbd658a880ec2efe3 2024-06-01T05:44:25.9796507Z deleted: sha256:7929f2d54eb354fb693e05f6c84ce90d1e057a82afbef54db7f231c00d83216a 2024-06-01T05:44:25.9797342Z deleted: sha256:42b6b0b889995e150471e5867e7b5db280aa40d2d157ed8d9483fee15b62f739 2024-06-01T05:44:25.9798189Z deleted: sha256:6a2d484294dab1506bd0dd581278e9bc85bf9a86f0a821994ce646fd5d1d7a00 2024-06-01T05:44:25.9799033Z deleted: sha256:8c2df0524784129b6d6896c6103132ce814d7478aca73c904e1bddbf78bea076 2024-06-01T05:44:25.9799882Z deleted: sha256:61f49ad2caef08555dd0b047a87f2076fdb7f9829c0422214c34abb43ca255ee 2024-06-01T05:44:25.9800727Z deleted: sha256:031c0450f806ba8240979dbfde7f012cce7f7f054d9f1863e918c5a86f43a2fe 2024-06-01T05:44:25.9801586Z deleted: sha256:15d84fd65a6a2fc897e2ca6201a4ce5279975a7266d6b7032af3baaf2e81f5b5 2024-06-01T05:44:25.9802554Z deleted: sha256:d6e3d0de25c9e7a022d14ff560ca05790c39fa83887ff1bfbbbe996d11d7af0b 2024-06-01T05:44:25.9803446Z deleted: sha256:21bf9cdc5a0d9cfdd3d9519bf2a23f8ad7280443171e8a90e0e40b303d219a58 2024-06-01T05:44:25.9804334Z deleted: sha256:56bedd909d4abc4c972f26bf25d60b228e07806f8ec9eefb1a1f7a049529077d 2024-06-01T05:44:25.9805189Z deleted: sha256:f845a36b8874f3440eada3908265f7bc8afd9067181b3923122979b0b74c156c 2024-06-01T05:44:25.9806053Z deleted: sha256:f49871f63a7f775620906ef5dcc1bf94fddd37caafa37985bdd748d88f1277fc 2024-06-01T05:44:25.9807034Z deleted: sha256:067856d12e70a5f18dd5201f5ad951c68b5f2af7d97f337b2a10f78a78455aa6 2024-06-01T05:44:25.9807919Z deleted: sha256:3c1db89ecbbad3c1d92a5fd2cb0f2c95acb0f062221bbd03d520346e3b89cca6 2024-06-01T05:44:25.9808782Z deleted: sha256:35f1dc195242d8a22c632a46b3d635ebf8a2e409cbe054f11c79592851c3fde6 2024-06-01T05:44:25.9809630Z deleted: sha256:59ae3c674c860573df0f0876aa25d7b5b3c1f6bbd3600ca76a48f30e351cf1b7 2024-06-01T05:44:25.9810488Z deleted: sha256:9e966a235307abbaa87d3ddf72cfa46594d2e74ab8e9a980d2bc604973297970 2024-06-01T05:44:25.9811352Z deleted: sha256:64e91944126efbbf31cb1c62d035210a9f77bbba41de3ef7b76d27dcb7962d68 2024-06-01T05:44:25.9812200Z deleted: sha256:85e4579e03101245d057d03eb06ba56ca9712f1dd6a8a18c7bca53f6974f6a88 2024-06-01T05:44:25.9813138Z deleted: sha256:431c1da34f7554e53e28d6bc0b8587ec5cdf6cecb9048c6973c3996928df571e 2024-06-01T05:44:25.9814019Z deleted: sha256:ddff7d53270d30040c4d7eb09bbfce0e2adfb15c42e9cd2bb437fa821c68ef45 2024-06-01T05:44:25.9814940Z deleted: sha256:b9925f8ba4cbf67cb1bed356a379b3db5a7cfa55da86e443bb23f53999bcce8f 2024-06-01T05:44:25.9815832Z deleted: sha256:7cc6eaea33417a03d56e448b55ff6a9ded4ce9b873939a20315f7d37202d3399 2024-06-01T05:44:25.9816690Z deleted: sha256:5f62642e3d17aa40b4a492e476747bb282fb1c38d5a583c9b359e6c1bcc0b20f 2024-06-01T05:44:25.9817569Z deleted: sha256:9fe5e52fddbfa52901cd08bc6faa9db9fa35f26e00a884a8dd72e23056768028 2024-06-01T05:44:25.9818446Z deleted: sha256:56bf8adc1a2f89eac613e15f91e6bca677c7889f91e02e85e2aaaf755d838333 2024-06-01T05:44:25.9819318Z deleted: sha256:c944b4681a5eae58a040ffdeea0acf939b94bdedfd0a7c69542698b83ac9e76f 2024-06-01T05:44:25.9820299Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053 2024-06-01T05:44:25.9821195Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6 2024-06-01T05:44:25.9822077Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737 2024-06-01T05:44:25.9822933Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732 2024-06-01T05:44:25.9823793Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9 2024-06-01T05:44:25.9824655Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650 2024-06-01T05:44:25.9825508Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe 2024-06-01T05:44:25.9826375Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24 2024-06-01T05:44:25.9827251Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c 2024-06-01T05:44:25.9828144Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855 2024-06-01T05:44:25.9829026Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c 2024-06-01T05:44:25.9829551Z 2024-06-01T05:44:25.9829709Z Total reclaimed space: 42.27GB 2024-06-01T05:44:25.9862723Z ##[group]Run set +e 2024-06-01T05:44:25.9863119Z set +e 2024-06-01T05:44:25.9863416Z set -x 2024-06-01T05:44:25.9863714Z  2024-06-01T05:44:25.9864001Z nvidia-smi 2024-06-01T05:44:25.9864641Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2024-06-01T05:44:25.9865652Z # the case where the driver has already crashed as it still can get the driver version 2024-06-01T05:44:25.9866647Z # and some basic information like the bus ID. However, the rest of the information 2024-06-01T05:44:25.9867409Z # would be missing (ERR!), for example: 2024-06-01T05:44:25.9867874Z # 2024-06-01T05:44:25.9868297Z # +-----------------------------------------------------------------------------+ 2024-06-01T05:44:25.9869086Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2024-06-01T05:44:25.9869892Z # |-------------------------------+----------------------+----------------------+ 2024-06-01T05:44:25.9870923Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-01T05:44:25.9871821Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2024-06-01T05:44:25.9872589Z # | | | MIG M. | 2024-06-01T05:44:25.9873192Z # |===============================+======================+======================| 2024-06-01T05:44:25.9873851Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2024-06-01T05:44:25.9874636Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2024-06-01T05:44:25.9875346Z # | | | ERR! | 2024-06-01T05:44:25.9875992Z # +-------------------------------+----------------------+----------------------+ 2024-06-01T05:44:25.9876526Z # 2024-06-01T05:44:25.9876951Z # +-----------------------------------------------------------------------------+ 2024-06-01T05:44:25.9877617Z # | Processes: | 2024-06-01T05:44:25.9878362Z # | GPU GI CI PID Type Process name GPU Memory | 2024-06-01T05:44:25.9879086Z # | ID ID Usage | 2024-06-01T05:44:25.9879712Z # |=============================================================================| 2024-06-01T05:44:25.9880473Z # +-----------------------------------------------------------------------------+ 2024-06-01T05:44:25.9880989Z # 2024-06-01T05:44:25.9881559Z # This should be reported as a failure instead as it will guarantee to fail when 2024-06-01T05:44:25.9882361Z # Docker tries to run with --gpus all 2024-06-01T05:44:25.9882813Z # 2024-06-01T05:44:25.9883361Z # So, the correct check here is to query one of the missing piece of info like 2024-06-01T05:44:25.9884161Z # GPU name, so that the command can fail accordingly 2024-06-01T05:44:25.9884873Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-01T05:44:25.9885468Z NVIDIA_SMI_STATUS=$? 2024-06-01T05:44:25.9885831Z  2024-06-01T05:44:25.9886464Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-06-01T05:44:25.9887407Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-06-01T05:44:25.9888275Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-06-01T05:44:25.9889006Z  .github/scripts/stop_runner_service.sh 2024-06-01T05:44:25.9889467Z fi 2024-06-01T05:44:25.9889751Z  2024-06-01T05:44:25.9890433Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2024-06-01T05:44:25.9891442Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2024-06-01T05:44:25.9892259Z # https://github.com/pytorch/test-infra/issues/4000 2024-06-01T05:44:25.9892874Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2024-06-01T05:44:25.9893365Z NVIDIA_SMI_STATUS=$? 2024-06-01T05:44:25.9893744Z  2024-06-01T05:44:25.9894373Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-06-01T05:44:25.9895319Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-06-01T05:44:25.9896177Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-06-01T05:44:25.9896912Z  .github/scripts/stop_runner_service.sh 2024-06-01T05:44:25.9897368Z fi 2024-06-01T05:44:25.9897656Z  2024-06-01T05:44:25.9898017Z # Check the GPU count to be a power of 2 2024-06-01T05:44:25.9898883Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2024-06-01T05:44:25.9900012Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2024-06-01T05:44:25.9900851Z  .github/scripts/stop_runner_service.sh 2024-06-01T05:44:25.9901318Z fi 2024-06-01T05:44:25.9908702Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:25.9909213Z env: 2024-06-01T05:44:25.9909485Z GIT_DEFAULT_BRANCH: main 2024-06-01T05:44:25.9910154Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-01T05:44:25.9910949Z DOCKER_CONTAINER_ID: 115e64e466b10a086a85f3fd11d1577e5c385bd10f36038626ba02a0bb738b94 2024-06-01T05:44:25.9911755Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2024-06-01T05:44:25.9912287Z ##[endgroup] 2024-06-01T05:44:25.9932850Z + nvidia-smi 2024-06-01T05:44:27.5665647Z Sat Jun 1 05:44:27 2024 2024-06-01T05:44:27.5666467Z +-----------------------------------------------------------------------------------------+ 2024-06-01T05:44:27.5667521Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-01T05:44:27.5668417Z |-----------------------------------------+------------------------+----------------------+ 2024-06-01T05:44:27.5669236Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-01T05:44:27.5670407Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-01T05:44:27.5671300Z | | | MIG M. | 2024-06-01T05:44:27.5671872Z |=========================================+========================+======================| 2024-06-01T05:44:27.5805371Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-01T05:44:27.5806230Z | 0% 27C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-06-01T05:44:27.5806875Z | | | N/A | 2024-06-01T05:44:27.5807568Z +-----------------------------------------+------------------------+----------------------+ 2024-06-01T05:44:27.5808186Z 2024-06-01T05:44:27.5808870Z +-----------------------------------------------------------------------------------------+ 2024-06-01T05:44:27.5809516Z | Processes: | 2024-06-01T05:44:27.5810246Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-01T05:44:27.5811093Z | ID ID Usage | 2024-06-01T05:44:27.5825075Z |=========================================================================================| 2024-06-01T05:44:27.5825809Z | No running processes found | 2024-06-01T05:44:27.5826616Z +-----------------------------------------------------------------------------------------+ 2024-06-01T05:44:28.1822805Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-01T05:44:29.7518637Z NVIDIA A10G 2024-06-01T05:44:30.1977383Z + NVIDIA_SMI_STATUS=0 2024-06-01T05:44:30.1977932Z + '[' 0 -ne 0 ']' 2024-06-01T05:44:30.1981996Z ++ nvidia-smi --list-gpus 2024-06-01T05:44:30.1982449Z ++ wc -l 2024-06-01T05:44:32.2103309Z + GPU_COUNT=1 2024-06-01T05:44:32.2103728Z + NVIDIA_SMI_STATUS=0 2024-06-01T05:44:32.2104361Z + '[' 0 -ne 0 ']' 2024-06-01T05:44:32.2104809Z + '[' 1 -le 8 ']' 2024-06-01T05:44:32.2105275Z + '[' 1 -ne 1 ']' 2024-06-01T05:44:32.2165893Z Post job cleanup. 2024-06-01T05:44:32.2215074Z Post job cleanup. 2024-06-01T05:44:32.3040485Z [command]/usr/bin/git version 2024-06-01T05:44:32.3075936Z git version 2.40.1 2024-06-01T05:44:32.3113645Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0db6a8d1-0846-4bb8-8a52-5cccf93f3b72' before making global git config changes 2024-06-01T05:44:32.3115034Z Adding repository directory to the temporary git global config as a safe directory 2024-06-01T05:44:32.3119078Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-01T05:44:32.3145475Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-01T05:44:32.3175374Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-01T05:44:32.3384640Z Entering 'android/libs/fbjni' 2024-06-01T05:44:32.3419717Z Entering 'third_party/FP16' 2024-06-01T05:44:32.3454311Z Entering 'third_party/FXdiv' 2024-06-01T05:44:32.3490937Z Entering 'third_party/NNPACK' 2024-06-01T05:44:32.3527674Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T05:44:32.3564389Z Entering 'third_party/XNNPACK' 2024-06-01T05:44:32.3609506Z Entering 'third_party/benchmark' 2024-06-01T05:44:32.3647676Z Entering 'third_party/cpp-httplib' 2024-06-01T05:44:32.3683837Z Entering 'third_party/cpuinfo' 2024-06-01T05:44:32.3717133Z Entering 'third_party/cudnn_frontend' 2024-06-01T05:44:32.3754472Z Entering 'third_party/cutlass' 2024-06-01T05:44:32.3795523Z Entering 'third_party/eigen' 2024-06-01T05:44:32.3830182Z Entering 'third_party/fbgemm' 2024-06-01T05:44:32.3865876Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T05:44:32.3899695Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T05:44:32.3933975Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T05:44:32.3971930Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T05:44:32.4005231Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T05:44:32.4039447Z Entering 'third_party/flatbuffers' 2024-06-01T05:44:32.4079706Z Entering 'third_party/fmt' 2024-06-01T05:44:32.4115447Z Entering 'third_party/foxi' 2024-06-01T05:44:32.4148659Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T05:44:32.4184213Z Entering 'third_party/gloo' 2024-06-01T05:44:32.4219370Z Entering 'third_party/googletest' 2024-06-01T05:44:32.4253729Z Entering 'third_party/ideep' 2024-06-01T05:44:32.4288283Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T05:44:32.4328761Z Entering 'third_party/ios-cmake' 2024-06-01T05:44:32.4365028Z Entering 'third_party/ittapi' 2024-06-01T05:44:32.4401283Z Entering 'third_party/kineto' 2024-06-01T05:44:32.4437392Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T05:44:32.4471849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T05:44:32.4506197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T05:44:32.4540861Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T05:44:32.4573598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T05:44:32.4608099Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T05:44:32.4645492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T05:44:32.4681776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T05:44:32.4714469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T05:44:32.4750331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T05:44:32.4785794Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T05:44:32.4820636Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T05:44:32.4856599Z Entering 'third_party/mimalloc' 2024-06-01T05:44:32.4891745Z Entering 'third_party/nccl/nccl' 2024-06-01T05:44:32.4927867Z Entering 'third_party/nlohmann' 2024-06-01T05:44:32.4962751Z Entering 'third_party/onnx' 2024-06-01T05:44:32.5010302Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T05:44:32.5045766Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T05:44:32.5082055Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T05:44:32.5118074Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T05:44:32.5152359Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T05:44:32.5184354Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T05:44:32.5218786Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T05:44:32.5254395Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T05:44:32.5286967Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T05:44:32.5320633Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T05:44:32.5355133Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T05:44:32.5390207Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T05:44:32.5425101Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T05:44:32.5475675Z Entering 'third_party/pocketfft' 2024-06-01T05:44:32.5509121Z Entering 'third_party/protobuf' 2024-06-01T05:44:32.5545210Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T05:44:32.5580354Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T05:44:32.5616462Z Entering 'third_party/psimd' 2024-06-01T05:44:32.5648719Z Entering 'third_party/pthreadpool' 2024-06-01T05:44:32.5683661Z Entering 'third_party/pybind11' 2024-06-01T05:44:32.5716266Z Entering 'third_party/python-peachpy' 2024-06-01T05:44:32.5751903Z Entering 'third_party/sleef' 2024-06-01T05:44:32.5786123Z Entering 'third_party/tensorpipe' 2024-06-01T05:44:32.5821035Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T05:44:32.5855470Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T05:44:32.5890901Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T05:44:32.5926199Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T05:44:32.5959996Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T05:44:32.6010513Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-01T05:44:32.6031300Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6040723Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-01T05:44:32.6068893Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-01T05:44:32.6271325Z Entering 'android/libs/fbjni' 2024-06-01T05:44:32.6294518Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6317103Z Entering 'third_party/FP16' 2024-06-01T05:44:32.6337916Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6362057Z Entering 'third_party/FXdiv' 2024-06-01T05:44:32.6383220Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6405276Z Entering 'third_party/NNPACK' 2024-06-01T05:44:32.6426928Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6449672Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-01T05:44:32.6470569Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6494830Z Entering 'third_party/XNNPACK' 2024-06-01T05:44:32.6516271Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6551343Z Entering 'third_party/benchmark' 2024-06-01T05:44:32.6572877Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6595980Z Entering 'third_party/cpp-httplib' 2024-06-01T05:44:32.6617835Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6640076Z Entering 'third_party/cpuinfo' 2024-06-01T05:44:32.6661453Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6685366Z Entering 'third_party/cudnn_frontend' 2024-06-01T05:44:32.6706871Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6729581Z Entering 'third_party/cutlass' 2024-06-01T05:44:32.6750067Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6778892Z Entering 'third_party/eigen' 2024-06-01T05:44:32.6801751Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6826796Z Entering 'third_party/fbgemm' 2024-06-01T05:44:32.6848804Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6872018Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-01T05:44:32.6892910Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6916071Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-01T05:44:32.6937105Z http.https://github.com/.extraheader 2024-06-01T05:44:32.6960082Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-01T05:44:32.6980980Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7008165Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-01T05:44:32.7028744Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7052418Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-01T05:44:32.7073408Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7096437Z Entering 'third_party/flatbuffers' 2024-06-01T05:44:32.7118240Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7143548Z Entering 'third_party/fmt' 2024-06-01T05:44:32.7166214Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7191048Z Entering 'third_party/foxi' 2024-06-01T05:44:32.7212472Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7234806Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-01T05:44:32.7255619Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7277791Z Entering 'third_party/gloo' 2024-06-01T05:44:32.7299316Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7322445Z Entering 'third_party/googletest' 2024-06-01T05:44:32.7344934Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7367793Z Entering 'third_party/ideep' 2024-06-01T05:44:32.7389221Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7412328Z Entering 'third_party/ideep/mkl-dnn' 2024-06-01T05:44:32.7433004Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7461011Z Entering 'third_party/ios-cmake' 2024-06-01T05:44:32.7483718Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7506155Z Entering 'third_party/ittapi' 2024-06-01T05:44:32.7527425Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7549879Z Entering 'third_party/kineto' 2024-06-01T05:44:32.7572079Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7596583Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-01T05:44:32.7617390Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7641582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-01T05:44:32.7663518Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7687256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-01T05:44:32.7707330Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7729603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-01T05:44:32.7751586Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7775268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-01T05:44:32.7796886Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7819135Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-01T05:44:32.7840311Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7865534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-01T05:44:32.7887548Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7910492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-01T05:44:32.7932303Z http.https://github.com/.extraheader 2024-06-01T05:44:32.7955832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-01T05:44:32.7976704Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8000325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-01T05:44:32.8020773Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8045069Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-01T05:44:32.8066100Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8089109Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-01T05:44:32.8108015Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8130458Z Entering 'third_party/mimalloc' 2024-06-01T05:44:32.8153121Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8174152Z Entering 'third_party/nccl/nccl' 2024-06-01T05:44:32.8198188Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8221946Z Entering 'third_party/nlohmann' 2024-06-01T05:44:32.8243433Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8266292Z Entering 'third_party/onnx' 2024-06-01T05:44:32.8287350Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8321654Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-01T05:44:32.8342576Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8365404Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-01T05:44:32.8386312Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8411967Z Entering 'third_party/opentelemetry-cpp' 2024-06-01T05:44:32.8434031Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8458169Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-01T05:44:32.8479442Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8501718Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-01T05:44:32.8523013Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8545538Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-01T05:44:32.8568361Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8591186Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-01T05:44:32.8612426Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8635004Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-01T05:44:32.8656104Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8678690Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-01T05:44:32.8699785Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8722229Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-01T05:44:32.8743221Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8767388Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-01T05:44:32.8787554Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8812695Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-01T05:44:32.8833519Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8857472Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-01T05:44:32.8878865Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8917600Z Entering 'third_party/pocketfft' 2024-06-01T05:44:32.8940070Z http.https://github.com/.extraheader 2024-06-01T05:44:32.8962303Z Entering 'third_party/protobuf' 2024-06-01T05:44:32.8983720Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9009620Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-01T05:44:32.9031015Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9052534Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-01T05:44:32.9074524Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9099280Z Entering 'third_party/psimd' 2024-06-01T05:44:32.9121131Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9144156Z Entering 'third_party/pthreadpool' 2024-06-01T05:44:32.9165279Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9187556Z Entering 'third_party/pybind11' 2024-06-01T05:44:32.9209206Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9231405Z Entering 'third_party/python-peachpy' 2024-06-01T05:44:32.9253886Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9277721Z Entering 'third_party/sleef' 2024-06-01T05:44:32.9298733Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9321761Z Entering 'third_party/tensorpipe' 2024-06-01T05:44:32.9342618Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9366185Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-01T05:44:32.9386653Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9409558Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-01T05:44:32.9431620Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9455242Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-01T05:44:32.9475581Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9498403Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-01T05:44:32.9519600Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9541983Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-01T05:44:32.9564300Z http.https://github.com/.extraheader 2024-06-01T05:44:32.9660413Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-01T05:44:32.9677955Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-01T05:44:32.9684521Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-01T05:44:32.9685046Z ##[endgroup] 2024-06-01T05:44:33.5975363Z Cleaning up orphan processes