2024-06-05T08:30:34.5628656Z Current runner version: '2.317.0'
2024-06-05T08:30:34.5634791Z Runner name: 'i-07bc38970f73de3fc'
2024-06-05T08:30:34.5635653Z Runner group name: 'Default'
2024-06-05T08:30:34.5636539Z Machine name: 'ip-10-0-62-168'
2024-06-05T08:30:34.5641262Z ##[group]GITHUB_TOKEN Permissions
2024-06-05T08:30:34.5643321Z Actions: read
2024-06-05T08:30:34.5643886Z Attestations: read
2024-06-05T08:30:34.5644393Z Checks: read
2024-06-05T08:30:34.5644837Z Contents: read
2024-06-05T08:30:34.5645344Z Deployments: read
2024-06-05T08:30:34.5645850Z Discussions: read
2024-06-05T08:30:34.5646403Z Issues: read
2024-06-05T08:30:34.5647128Z Metadata: read
2024-06-05T08:30:34.5647614Z Packages: read
2024-06-05T08:30:34.5648126Z Pages: read
2024-06-05T08:30:34.5648585Z PullRequests: read
2024-06-05T08:30:34.5649121Z RepositoryProjects: read
2024-06-05T08:30:34.5649762Z SecurityEvents: read
2024-06-05T08:30:34.5650278Z Statuses: read
2024-06-05T08:30:34.5650738Z ##[endgroup]
2024-06-05T08:30:34.5654010Z Secret source: Actions
2024-06-05T08:30:34.5654786Z Prepare workflow directory
2024-06-05T08:30:34.9229844Z Prepare all required actions
2024-06-05T08:30:34.9404856Z Getting action download info
2024-06-05T08:30:35.1105118Z Download action repository 'pytorch/test-infra@main' (SHA:ac44e13bbae4ae4d707caa21aaf9dd2f6c27b7da)
2024-06-05T08:30:35.4000552Z Download action repository 'pytorch/pytorch@main' (SHA:bb2de3b10120f91afce8da6233094076713f673d)
2024-06-05T08:30:38.1583475Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-06-05T08:30:38.2977277Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-06-05T08:30:38.5741656Z Getting action download info
2024-06-05T08:30:38.6918208Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-06-05T08:30:38.8648079Z Getting action download info
2024-06-05T08:30:38.9638686Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-06-05T08:30:39.0991690Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/127669 (dffed71f3397e435f3656f25960a4d75ad415746)
2024-06-05T08:30:39.0993987Z ##[group] Inputs
2024-06-05T08:30:39.0994504Z   build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:30:39.1003658Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-06-05T08:30:39.1013323Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:30:39.1014557Z   sync-tag: 
2024-06-05T08:30:39.1015484Z   timeout-minutes: 240
2024-06-05T08:30:39.1015859Z   use-gha: 
2024-06-05T08:30:39.1016186Z   dashboard-tag: 
2024-06-05T08:30:39.1016555Z   s3-bucket: gha-artifacts
2024-06-05T08:30:39.1016958Z   aws-role-to-assume: 
2024-06-05T08:30:39.1017330Z ##[endgroup]
2024-06-05T08:30:39.1018269Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:30:39.1608614Z A job started hook has been configured by the self-hosted runner administrator
2024-06-05T08:30:39.1747426Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh'
2024-06-05T08:30:39.1757996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:30:39.1758567Z ##[endgroup]
2024-06-05T08:30:40.6328085Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2024-06-05T08:30:40.6328771Z with:
2024-06-05T08:30:40.6329579Z   github-secret: ***
2024-06-05T08:30:40.6330673Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-06-05T08:30:40.6331810Z   activate-with-label: false
2024-06-05T08:30:40.6332290Z   label: with-ssh
2024-06-05T08:30:40.6332718Z   remove-existing-keys: true
2024-06-05T08:30:40.6333182Z env:
2024-06-05T08:30:40.6333567Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:30:40.6334020Z ##[endgroup]
2024-06-05T08:30:40.7206128Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-06-05T08:30:40.7207800Z ciflow reference detected, attempting to extract PR number
2024-06-05T08:30:41.0826896Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2024-06-05T08:30:41.1409639Z No SSH keys found for user pytorch-bot[bot]
2024-06-05T08:30:41.1410313Z Grabbing public ssh keys from https://github.com/Fuzzkatt.keys
2024-06-05T08:30:41.2142048Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2024-06-05T08:30:41.2155702Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2024-06-05T08:30:41.2180442Z Login using: ssh ec2-user@ec2-3-82-48-118.compute-1.amazonaws.com
2024-06-05T08:30:41.2181270Z All testing is done inside the container, to start an interactive session run:
2024-06-05T08:30:41.2182126Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2024-06-05T08:30:41.2286880Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2024-06-05T08:30:41.2287499Z with:
2024-06-05T08:30:41.2287810Z   submodules: recursive
2024-06-05T08:30:41.2288175Z   fetch-depth: 0
2024-06-05T08:30:41.2288504Z env:
2024-06-05T08:30:41.2288806Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:30:41.2289175Z ##[endgroup]
2024-06-05T08:30:41.2508707Z ##[group]Run retry () {
2024-06-05T08:30:41.2509135Z retry () {
2024-06-05T08:30:41.2509699Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-06-05T08:30:41.2510300Z }
2024-06-05T08:30:41.2510647Z echo "${GITHUB_WORKSPACE}"
2024-06-05T08:30:41.2511100Z if [ -z "${NO_SUDO}" ]; then
2024-06-05T08:30:41.2511622Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-06-05T08:30:41.2512116Z else
2024-06-05T08:30:41.2512497Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-06-05T08:30:41.2512960Z fi
2024-06-05T08:30:41.2513344Z mkdir "${GITHUB_WORKSPACE}"
2024-06-05T08:30:41.2521456Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:30:41.2522168Z env:
2024-06-05T08:30:41.2522488Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:30:41.2522869Z   NO_SUDO: 
2024-06-05T08:30:41.2523168Z ##[endgroup]
2024-06-05T08:30:41.2545061Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T08:30:43.9620873Z ##[group]Run malfet/checkout@silent-checkout
2024-06-05T08:30:43.9621347Z with:
2024-06-05T08:30:43.9621707Z   ref: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:30:43.9622189Z   fetch-depth: 0
2024-06-05T08:30:43.9622535Z   submodules: recursive
2024-06-05T08:30:43.9622904Z   quiet-checkout: true
2024-06-05T08:30:43.9623288Z   repository: pytorch/pytorch
2024-06-05T08:30:43.9623804Z   token: ***
2024-06-05T08:30:43.9624129Z   ssh-strict: true
2024-06-05T08:30:43.9624484Z   persist-credentials: true
2024-06-05T08:30:43.9624874Z   clean: true
2024-06-05T08:30:43.9625241Z   sparse-checkout-cone-mode: true
2024-06-05T08:30:43.9625675Z   lfs: false
2024-06-05T08:30:43.9626001Z   set-safe-directory: true
2024-06-05T08:30:43.9626371Z env:
2024-06-05T08:30:43.9626675Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:30:43.9627042Z ##[endgroup]
2024-06-05T08:30:44.0545030Z Syncing repository: pytorch/pytorch
2024-06-05T08:30:44.0546846Z ##[group]Getting Git version info
2024-06-05T08:30:44.0547767Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-06-05T08:30:44.0548715Z [command]/usr/bin/git version
2024-06-05T08:30:44.0549130Z git version 2.40.1
2024-06-05T08:30:44.0550732Z ##[endgroup]
2024-06-05T08:30:44.0563237Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/41387544-d205-4712-a526-726e8950dc69/.gitconfig'
2024-06-05T08:30:44.0564979Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/41387544-d205-4712-a526-726e8950dc69' before making global git config changes
2024-06-05T08:30:44.0566344Z Adding repository directory to the temporary git global config as a safe directory
2024-06-05T08:30:44.0567739Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T08:30:44.0592688Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-06-05T08:30:44.0596267Z ##[group]Initializing the repository
2024-06-05T08:30:44.0599061Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T08:30:44.0624029Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-06-05T08:30:44.0625241Z hint: is subject to change. To configure the initial branch name to use in all
2024-06-05T08:30:44.0626140Z hint: of your new repositories, which will suppress this warning, call:
2024-06-05T08:30:44.0626980Z hint: 
2024-06-05T08:30:44.0627472Z hint: 	git config --global init.defaultBranch <name>
2024-06-05T08:30:44.0628029Z hint: 
2024-06-05T08:30:44.0628813Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-06-05T08:30:44.0629780Z hint: 'development'. The just-created branch can be renamed via this command:
2024-06-05T08:30:44.0630434Z hint: 
2024-06-05T08:30:44.0630795Z hint: 	git branch -m <name>
2024-06-05T08:30:44.0631609Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-06-05T08:30:44.0633220Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-06-05T08:30:44.0658320Z ##[endgroup]
2024-06-05T08:30:44.0658986Z ##[group]Disabling automatic garbage collection
2024-06-05T08:30:44.0660270Z [command]/usr/bin/git config --local gc.auto 0
2024-06-05T08:30:44.0683505Z ##[endgroup]
2024-06-05T08:30:44.0684088Z ##[group]Setting up auth
2024-06-05T08:30:44.0689703Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-06-05T08:30:44.0714530Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-06-05T08:30:44.0916011Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-06-05T08:30:44.0938515Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-06-05T08:30:44.1137407Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-06-05T08:30:44.1174524Z ##[endgroup]
2024-06-05T08:30:44.1175136Z ##[group]Fetching the repository
2024-06-05T08:30:44.1180548Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-06-05T08:30:47.1492006Z remote: Enumerating objects: 1114661        
2024-06-05T08:30:47.1492634Z remote: Enumerating objects: 1118742, done.        
2024-06-05T08:30:47.1496526Z remote: Counting objects:   0% (1/4081)        
2024-06-05T08:30:47.1499041Z remote: Counting objects:   1% (41/4081)        
2024-06-05T08:30:47.1501479Z remote: Counting objects:   2% (82/4081)        
2024-06-05T08:30:47.1504013Z remote: Counting objects:   3% (123/4081)        
2024-06-05T08:30:47.1506600Z remote: Counting objects:   4% (164/4081)        
2024-06-05T08:30:47.1509150Z remote: Counting objects:   5% (205/4081)        
2024-06-05T08:30:47.1511574Z remote: Counting objects:   6% (245/4081)        
2024-06-05T08:30:47.1512149Z remote: Counting objects:   7% (286/4081)        
2024-06-05T08:30:47.1514802Z remote: Counting objects:   8% (327/4081)        
2024-06-05T08:30:47.1515430Z remote: Counting objects:   9% (368/4081)        
2024-06-05T08:30:47.1516488Z remote: Counting objects:  10% (409/4081)        
2024-06-05T08:30:47.1517078Z remote: Counting objects:  11% (449/4081)        
2024-06-05T08:30:47.1521869Z remote: Counting objects:  12% (490/4081)        
2024-06-05T08:30:47.1528785Z remote: Counting objects:  13% (531/4081)        
2024-06-05T08:30:47.1529440Z remote: Counting objects:  14% (572/4081)        
2024-06-05T08:30:47.1530057Z remote: Counting objects:  15% (613/4081)        
2024-06-05T08:30:47.1530617Z remote: Counting objects:  16% (653/4081)        
2024-06-05T08:30:47.1531166Z remote: Counting objects:  17% (694/4081)        
2024-06-05T08:30:47.1531725Z remote: Counting objects:  18% (735/4081)        
2024-06-05T08:30:47.1532291Z remote: Counting objects:  19% (776/4081)        
2024-06-05T08:30:47.1532847Z remote: Counting objects:  20% (817/4081)        
2024-06-05T08:30:47.1533396Z remote: Counting objects:  21% (858/4081)        
2024-06-05T08:30:47.1533961Z remote: Counting objects:  22% (898/4081)        
2024-06-05T08:30:47.1536815Z remote: Counting objects:  23% (939/4081)        
2024-06-05T08:30:47.1537453Z remote: Counting objects:  24% (980/4081)        
2024-06-05T08:30:47.1538047Z remote: Counting objects:  25% (1021/4081)        
2024-06-05T08:30:47.1539029Z remote: Counting objects:  26% (1062/4081)        
2024-06-05T08:30:47.1539690Z remote: Counting objects:  27% (1102/4081)        
2024-06-05T08:30:47.1540327Z remote: Counting objects:  28% (1143/4081)        
2024-06-05T08:30:47.1540941Z remote: Counting objects:  29% (1184/4081)        
2024-06-05T08:30:47.1541526Z remote: Counting objects:  30% (1225/4081)        
2024-06-05T08:30:47.1542106Z remote: Counting objects:  31% (1266/4081)        
2024-06-05T08:30:47.1542862Z remote: Counting objects:  32% (1306/4081)        
2024-06-05T08:30:47.1543511Z remote: Counting objects:  33% (1347/4081)        
2024-06-05T08:30:47.1544155Z remote: Counting objects:  34% (1388/4081)        
2024-06-05T08:30:47.1544772Z remote: Counting objects:  35% (1429/4081)        
2024-06-05T08:30:47.1545518Z remote: Counting objects:  36% (1470/4081)        
2024-06-05T08:30:47.1546147Z remote: Counting objects:  37% (1510/4081)        
2024-06-05T08:30:47.1546880Z remote: Counting objects:  38% (1551/4081)        
2024-06-05T08:30:47.1547566Z remote: Counting objects:  39% (1592/4081)        
2024-06-05T08:30:47.1548469Z remote: Counting objects:  40% (1633/4081)        
2024-06-05T08:30:47.1549046Z remote: Counting objects:  41% (1674/4081)        
2024-06-05T08:30:47.1549623Z remote: Counting objects:  42% (1715/4081)        
2024-06-05T08:30:47.1550201Z remote: Counting objects:  43% (1755/4081)        
2024-06-05T08:30:47.1550777Z remote: Counting objects:  44% (1796/4081)        
2024-06-05T08:30:47.1551348Z remote: Counting objects:  45% (1837/4081)        
2024-06-05T08:30:47.1551980Z remote: Counting objects:  46% (1878/4081)        
2024-06-05T08:30:47.1552602Z remote: Counting objects:  47% (1919/4081)        
2024-06-05T08:30:47.1553168Z remote: Counting objects:  48% (1959/4081)        
2024-06-05T08:30:47.1553729Z remote: Counting objects:  49% (2000/4081)        
2024-06-05T08:30:47.1554293Z remote: Counting objects:  50% (2041/4081)        
2024-06-05T08:30:47.1554862Z remote: Counting objects:  51% (2082/4081)        
2024-06-05T08:30:47.1555434Z remote: Counting objects:  52% (2123/4081)        
2024-06-05T08:30:47.1555995Z remote: Counting objects:  53% (2163/4081)        
2024-06-05T08:30:47.1556588Z remote: Counting objects:  54% (2204/4081)        
2024-06-05T08:30:47.1557156Z remote: Counting objects:  55% (2245/4081)        
2024-06-05T08:30:47.1557726Z remote: Counting objects:  56% (2286/4081)        
2024-06-05T08:30:47.1558298Z remote: Counting objects:  57% (2327/4081)        
2024-06-05T08:30:47.1558864Z remote: Counting objects:  58% (2367/4081)        
2024-06-05T08:30:47.1559434Z remote: Counting objects:  59% (2408/4081)        
2024-06-05T08:30:47.1560000Z remote: Counting objects:  60% (2449/4081)        
2024-06-05T08:30:47.1560568Z remote: Counting objects:  61% (2490/4081)        
2024-06-05T08:30:47.1561129Z remote: Counting objects:  62% (2531/4081)        
2024-06-05T08:30:47.1561842Z remote: Counting objects:  63% (2572/4081)        
2024-06-05T08:30:47.1562434Z remote: Counting objects:  64% (2612/4081)        
2024-06-05T08:30:47.1563015Z remote: Counting objects:  65% (2653/4081)        
2024-06-05T08:30:47.1563592Z remote: Counting objects:  66% (2694/4081)        
2024-06-05T08:30:47.1564164Z remote: Counting objects:  67% (2735/4081)        
2024-06-05T08:30:47.1564739Z remote: Counting objects:  68% (2776/4081)        
2024-06-05T08:30:47.1565315Z remote: Counting objects:  69% (2816/4081)        
2024-06-05T08:30:47.1565895Z remote: Counting objects:  70% (2857/4081)        
2024-06-05T08:30:47.1566636Z remote: Counting objects:  71% (2898/4081)        
2024-06-05T08:30:47.1567213Z remote: Counting objects:  72% (2939/4081)        
2024-06-05T08:30:47.1567784Z remote: Counting objects:  73% (2980/4081)        
2024-06-05T08:30:47.1568350Z remote: Counting objects:  74% (3020/4081)        
2024-06-05T08:30:47.1568923Z remote: Counting objects:  75% (3061/4081)        
2024-06-05T08:30:47.1569495Z remote: Counting objects:  76% (3102/4081)        
2024-06-05T08:30:47.1570171Z remote: Counting objects:  77% (3143/4081)        
2024-06-05T08:30:47.1570749Z remote: Counting objects:  78% (3184/4081)        
2024-06-05T08:30:47.1571325Z remote: Counting objects:  79% (3224/4081)        
2024-06-05T08:30:47.1571901Z remote: Counting objects:  80% (3265/4081)        
2024-06-05T08:30:47.1572476Z remote: Counting objects:  81% (3306/4081)        
2024-06-05T08:30:47.1573047Z remote: Counting objects:  82% (3347/4081)        
2024-06-05T08:30:47.1573624Z remote: Counting objects:  83% (3388/4081)        
2024-06-05T08:30:47.1574201Z remote: Counting objects:  84% (3429/4081)        
2024-06-05T08:30:47.1574776Z remote: Counting objects:  85% (3469/4081)        
2024-06-05T08:30:47.1575344Z remote: Counting objects:  86% (3510/4081)        
2024-06-05T08:30:47.1575918Z remote: Counting objects:  87% (3551/4081)        
2024-06-05T08:30:47.1576494Z remote: Counting objects:  88% (3592/4081)        
2024-06-05T08:30:47.1577068Z remote: Counting objects:  89% (3633/4081)        
2024-06-05T08:30:47.1577643Z remote: Counting objects:  90% (3673/4081)        
2024-06-05T08:30:47.1578296Z remote: Counting objects:  91% (3714/4081)        
2024-06-05T08:30:47.1578878Z remote: Counting objects:  92% (3755/4081)        
2024-06-05T08:30:47.1579457Z remote: Counting objects:  93% (3796/4081)        
2024-06-05T08:30:47.1580025Z remote: Counting objects:  94% (3837/4081)        
2024-06-05T08:30:47.1580597Z remote: Counting objects:  95% (3877/4081)        
2024-06-05T08:30:47.1581172Z remote: Counting objects:  96% (3918/4081)        
2024-06-05T08:30:47.1581745Z remote: Counting objects:  97% (3959/4081)        
2024-06-05T08:30:47.1582313Z remote: Counting objects:  98% (4000/4081)        
2024-06-05T08:30:47.1582890Z remote: Counting objects:  99% (4041/4081)        
2024-06-05T08:30:47.1583464Z remote: Counting objects: 100% (4081/4081)        
2024-06-05T08:30:47.1584087Z remote: Counting objects: 100% (4081/4081), done.        
2024-06-05T08:30:47.2199341Z remote: Compressing objects:   0% (1/1862)        
2024-06-05T08:30:47.2960119Z remote: Compressing objects:   1% (19/1862)        
2024-06-05T08:30:47.3122576Z remote: Compressing objects:   2% (38/1862)        
2024-06-05T08:30:47.3610483Z remote: Compressing objects:   3% (56/1862)        
2024-06-05T08:30:47.4974306Z remote: Compressing objects:   4% (75/1862)        
2024-06-05T08:30:47.5891204Z remote: Compressing objects:   5% (94/1862)        
2024-06-05T08:30:47.6598656Z remote: Compressing objects:   6% (112/1862)        
2024-06-05T08:30:47.6967707Z remote: Compressing objects:   7% (131/1862)        
2024-06-05T08:30:47.7442788Z remote: Compressing objects:   8% (149/1862)        
2024-06-05T08:30:47.7828626Z remote: Compressing objects:   9% (168/1862)        
2024-06-05T08:30:47.8124319Z remote: Compressing objects:  10% (187/1862)        
2024-06-05T08:30:47.8386778Z remote: Compressing objects:  11% (205/1862)        
2024-06-05T08:30:47.8553437Z remote: Compressing objects:  12% (224/1862)        
2024-06-05T08:30:47.8703507Z remote: Compressing objects:  13% (243/1862)        
2024-06-05T08:30:47.8789503Z remote: Compressing objects:  14% (261/1862)        
2024-06-05T08:30:47.8893010Z remote: Compressing objects:  15% (280/1862)        
2024-06-05T08:30:47.8948833Z remote: Compressing objects:  16% (298/1862)        
2024-06-05T08:30:47.8987021Z remote: Compressing objects:  17% (317/1862)        
2024-06-05T08:30:47.9012019Z remote: Compressing objects:  18% (336/1862)        
2024-06-05T08:30:47.9020549Z remote: Compressing objects:  19% (354/1862)        
2024-06-05T08:30:47.9061173Z remote: Compressing objects:  20% (373/1862)        
2024-06-05T08:30:47.9114298Z remote: Compressing objects:  21% (392/1862)        
2024-06-05T08:30:47.9143063Z remote: Compressing objects:  22% (410/1862)        
2024-06-05T08:30:47.9153819Z remote: Compressing objects:  23% (429/1862)        
2024-06-05T08:30:47.9165142Z remote: Compressing objects:  24% (447/1862)        
2024-06-05T08:30:47.9183746Z remote: Compressing objects:  25% (466/1862)        
2024-06-05T08:30:47.9202826Z remote: Compressing objects:  26% (485/1862)        
2024-06-05T08:30:47.9241632Z remote: Compressing objects:  27% (503/1862)        
2024-06-05T08:30:47.9247021Z remote: Compressing objects:  28% (522/1862)        
2024-06-05T08:30:47.9264118Z remote: Compressing objects:  29% (540/1862)        
2024-06-05T08:30:47.9285640Z remote: Compressing objects:  30% (559/1862)        
2024-06-05T08:30:47.9305895Z remote: Compressing objects:  31% (578/1862)        
2024-06-05T08:30:47.9335422Z remote: Compressing objects:  32% (596/1862)        
2024-06-05T08:30:47.9355158Z remote: Compressing objects:  33% (615/1862)        
2024-06-05T08:30:47.9370863Z remote: Compressing objects:  34% (634/1862)        
2024-06-05T08:30:47.9402315Z remote: Compressing objects:  35% (652/1862)        
2024-06-05T08:30:47.9423048Z remote: Compressing objects:  36% (671/1862)        
2024-06-05T08:30:47.9444580Z remote: Compressing objects:  37% (689/1862)        
2024-06-05T08:30:47.9466693Z remote: Compressing objects:  38% (708/1862)        
2024-06-05T08:30:47.9473668Z remote: Compressing objects:  39% (727/1862)        
2024-06-05T08:30:47.9482191Z remote: Compressing objects:  40% (745/1862)        
2024-06-05T08:30:47.9492525Z remote: Compressing objects:  41% (764/1862)        
2024-06-05T08:30:47.9498243Z remote: Compressing objects:  42% (783/1862)        
2024-06-05T08:30:47.9514715Z remote: Compressing objects:  43% (801/1862)        
2024-06-05T08:30:47.9533172Z remote: Compressing objects:  44% (820/1862)        
2024-06-05T08:30:47.9540409Z remote: Compressing objects:  45% (838/1862)        
2024-06-05T08:30:47.9558018Z remote: Compressing objects:  46% (857/1862)        
2024-06-05T08:30:47.9564176Z remote: Compressing objects:  47% (876/1862)        
2024-06-05T08:30:47.9570843Z remote: Compressing objects:  48% (894/1862)        
2024-06-05T08:30:47.9576813Z remote: Compressing objects:  49% (913/1862)        
2024-06-05T08:30:47.9580939Z remote: Compressing objects:  50% (931/1862)        
2024-06-05T08:30:47.9586728Z remote: Compressing objects:  51% (950/1862)        
2024-06-05T08:30:47.9591103Z remote: Compressing objects:  52% (969/1862)        
2024-06-05T08:30:47.9596550Z remote: Compressing objects:  53% (987/1862)        
2024-06-05T08:30:47.9598864Z remote: Compressing objects:  54% (1006/1862)        
2024-06-05T08:30:47.9601393Z remote: Compressing objects:  55% (1025/1862)        
2024-06-05T08:30:47.9602104Z remote: Compressing objects:  56% (1043/1862)        
2024-06-05T08:30:47.9604308Z remote: Compressing objects:  57% (1062/1862)        
2024-06-05T08:30:47.9604926Z remote: Compressing objects:  58% (1080/1862)        
2024-06-05T08:30:47.9605530Z remote: Compressing objects:  59% (1099/1862)        
2024-06-05T08:30:47.9606780Z remote: Compressing objects:  60% (1118/1862)        
2024-06-05T08:30:47.9609683Z remote: Compressing objects:  61% (1136/1862)        
2024-06-05T08:30:47.9610299Z remote: Compressing objects:  62% (1155/1862)        
2024-06-05T08:30:47.9610909Z remote: Compressing objects:  63% (1174/1862)        
2024-06-05T08:30:47.9611514Z remote: Compressing objects:  64% (1192/1862)        
2024-06-05T08:30:47.9612121Z remote: Compressing objects:  65% (1211/1862)        
2024-06-05T08:30:47.9612733Z remote: Compressing objects:  66% (1229/1862)        
2024-06-05T08:30:47.9618883Z remote: Compressing objects:  67% (1248/1862)        
2024-06-05T08:30:47.9629654Z remote: Compressing objects:  68% (1267/1862)        
2024-06-05T08:30:47.9636934Z remote: Compressing objects:  69% (1285/1862)        
2024-06-05T08:30:47.9642687Z remote: Compressing objects:  70% (1304/1862)        
2024-06-05T08:30:47.9648301Z remote: Compressing objects:  71% (1323/1862)        
2024-06-05T08:30:47.9655646Z remote: Compressing objects:  72% (1341/1862)        
2024-06-05T08:30:47.9663025Z remote: Compressing objects:  73% (1360/1862)        
2024-06-05T08:30:47.9670092Z remote: Compressing objects:  74% (1378/1862)        
2024-06-05T08:30:47.9672979Z remote: Compressing objects:  75% (1397/1862)        
2024-06-05T08:30:47.9675333Z remote: Compressing objects:  76% (1416/1862)        
2024-06-05T08:30:47.9689940Z remote: Compressing objects:  77% (1434/1862)        
2024-06-05T08:30:47.9692353Z remote: Compressing objects:  78% (1453/1862)        
2024-06-05T08:30:47.9692969Z remote: Compressing objects:  79% (1471/1862)        
2024-06-05T08:30:47.9693572Z remote: Compressing objects:  80% (1490/1862)        
2024-06-05T08:30:47.9694591Z remote: Compressing objects:  81% (1509/1862)        
2024-06-05T08:30:47.9695186Z remote: Compressing objects:  82% (1527/1862)        
2024-06-05T08:30:47.9698545Z remote: Compressing objects:  83% (1546/1862)        
2024-06-05T08:30:47.9702115Z remote: Compressing objects:  84% (1565/1862)        
2024-06-05T08:30:47.9706558Z remote: Compressing objects:  85% (1583/1862)        
2024-06-05T08:30:47.9711050Z remote: Compressing objects:  86% (1602/1862)        
2024-06-05T08:30:47.9711663Z remote: Compressing objects:  87% (1620/1862)        
2024-06-05T08:30:47.9712262Z remote: Compressing objects:  88% (1639/1862)        
2024-06-05T08:30:47.9715865Z remote: Compressing objects:  89% (1658/1862)        
2024-06-05T08:30:47.9716475Z remote: Compressing objects:  90% (1676/1862)        
2024-06-05T08:30:47.9718325Z remote: Compressing objects:  91% (1695/1862)        
2024-06-05T08:30:47.9718927Z remote: Compressing objects:  92% (1714/1862)        
2024-06-05T08:30:47.9720888Z remote: Compressing objects:  93% (1732/1862)        
2024-06-05T08:30:47.9723626Z remote: Compressing objects:  94% (1751/1862)        
2024-06-05T08:30:47.9724247Z remote: Compressing objects:  95% (1769/1862)        
2024-06-05T08:30:47.9726127Z remote: Compressing objects:  96% (1788/1862)        
2024-06-05T08:30:47.9728728Z remote: Compressing objects:  97% (1807/1862)        
2024-06-05T08:30:47.9731460Z remote: Compressing objects:  98% (1825/1862)        
2024-06-05T08:30:47.9732079Z remote: Compressing objects:  99% (1844/1862)        
2024-06-05T08:30:47.9733912Z remote: Compressing objects: 100% (1862/1862)        
2024-06-05T08:30:47.9734585Z remote: Compressing objects: 100% (1862/1862), done.        
2024-06-05T08:31:10.3719862Z remote: Total 1118742 (delta 2834), reused 3210 (delta 2213), pack-reused 1114661        
2024-06-05T08:31:33.8296244Z [command]/usr/bin/git rev-parse --verify --quiet dffed71f3397e435f3656f25960a4d75ad415746^{object}
2024-06-05T08:31:33.8315617Z dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:31:33.8319433Z ##[endgroup]
2024-06-05T08:31:33.8323525Z ##[group]Determining the checkout info
2024-06-05T08:31:33.8324780Z ##[endgroup]
2024-06-05T08:31:33.8325669Z ##[group]Checking out the ref
2024-06-05T08:31:33.8327010Z [command]/usr/bin/git checkout --quiet --force dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:31:35.0577877Z ##[endgroup]
2024-06-05T08:31:35.0578679Z ##[group]Setting up auth for fetching submodules
2024-06-05T08:31:35.0582180Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-06-05T08:31:35.0624317Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-06-05T08:31:35.0649914Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-06-05T08:31:35.0669398Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-06-05T08:31:35.0686195Z ##[endgroup]
2024-06-05T08:31:35.0686976Z ##[group]Fetching submodules
2024-06-05T08:31:35.0690674Z [command]/usr/bin/git submodule sync --recursive
2024-06-05T08:31:35.0896785Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-06-05T08:31:35.1099951Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-06-05T08:31:35.1101640Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-06-05T08:31:35.1103988Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-06-05T08:31:35.1105655Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-06-05T08:31:35.1107612Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-06-05T08:31:35.1109501Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-06-05T08:31:35.1111048Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-06-05T08:31:35.1112633Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-06-05T08:31:35.1114249Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-06-05T08:31:35.1116606Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-06-05T08:31:35.1118629Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-06-05T08:31:35.1121310Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-06-05T08:31:35.1123932Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-06-05T08:31:35.1127097Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-06-05T08:31:35.1129741Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-06-05T08:31:35.1132418Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2024-06-05T08:31:35.1135322Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:31:35.1137926Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-06-05T08:31:35.1140821Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-06-05T08:31:35.1143570Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-06-05T08:31:35.1146680Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-06-05T08:31:35.1149808Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-06-05T08:31:35.1153011Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-06-05T08:31:35.1156152Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-06-05T08:31:35.1159302Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-06-05T08:31:35.1162823Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-06-05T08:31:35.1167328Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-06-05T08:31:35.1170950Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-06-05T08:31:35.1174562Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-06-05T08:31:35.1178253Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-06-05T08:31:35.1181974Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-06-05T08:31:35.1185495Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-06-05T08:31:35.1189341Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-06-05T08:31:35.1192994Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-06-05T08:31:35.1196989Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-06-05T08:31:35.1217233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-06-05T08:31:35.3517355Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-06-05T08:31:35.5161245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-06-05T08:31:35.6781151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-06-05T08:31:35.9040810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-06-05T08:31:37.7847698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-06-05T08:31:47.6630213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-06-05T08:31:48.0403767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-06-05T08:31:48.5789019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-06-05T08:31:49.1542806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-06-05T08:31:50.3381983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-06-05T08:31:52.0481679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-06-05T08:31:57.7752109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-06-05T08:31:58.9121867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-06-05T08:32:00.6082438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-06-05T08:32:01.8451385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2024-06-05T08:32:02.0049952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-06-05T08:32:02.4139986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-06-05T08:32:02.7879860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-06-05T08:32:03.7386517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-06-05T08:32:04.0664405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-06-05T08:32:04.3160421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-06-05T08:32:05.7297711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-06-05T08:32:06.4783849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-06-05T08:32:07.2007114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-06-05T08:32:12.7683762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-06-05T08:32:15.1923911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-06-05T08:32:19.7329462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-06-05T08:32:19.9268416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-06-05T08:32:28.0825339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-06-05T08:32:28.2594574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-06-05T08:32:28.4790176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-06-05T08:32:29.4039922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-06-05T08:32:29.7271957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-06-05T08:32:30.3300483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-06-05T08:32:30.7699659Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-06-05T08:32:30.7781853Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-06-05T08:32:30.7842906Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-06-05T08:32:30.8028464Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-06-05T08:32:30.8336951Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-06-05T08:32:31.6127124Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2'
2024-06-05T08:32:31.6300885Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-06-05T08:32:31.6635943Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-06-05T08:32:31.7451933Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7'
2024-06-05T08:32:31.7704088Z Submodule path 'third_party/cudnn_frontend': checked out 'b740542818f36857acf7f9853f749bbad4118c65'
2024-06-05T08:32:32.2029315Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-06-05T08:32:32.4164049Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-06-05T08:32:32.4782956Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-06-05T08:32:32.4795110Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:32:32.4798168Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:32:32.4800112Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:32:32.4802201Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:32:32.4804069Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:32:32.4821419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-06-05T08:32:33.4613200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-06-05T08:32:34.0494316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-06-05T08:32:35.7710899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-06-05T08:32:36.7742065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-06-05T08:32:37.0908823Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-06-05T08:32:37.1742919Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-06-05T08:32:37.5175960Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-06-05T08:32:37.5717682Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-06-05T08:32:37.5808272Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-06-05T08:32:37.6744414Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-06-05T08:32:37.7056507Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281'
2024-06-05T08:32:37.7117880Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2024-06-05T08:32:37.7418814Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-06-05T08:32:37.7613218Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-06-05T08:32:37.7974513Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-06-05T08:32:37.8065361Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b'
2024-06-05T08:32:37.8074610Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-06-05T08:32:37.8090371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-06-05T08:32:49.8436030Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67'
2024-06-05T08:32:49.8574950Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-06-05T08:32:49.9355963Z Submodule path 'third_party/kineto': checked out 'be1317644c68b4bfc4646024a6b221066e430031'
2024-06-05T08:32:49.9398209Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:32:49.9400901Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:32:49.9403386Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:32:49.9405255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-06-05T08:32:50.4477710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-06-05T08:32:51.6092304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-06-05T08:32:52.6799356Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-06-05T08:32:52.6810726Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:32:52.6812624Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:32:52.6814478Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:32:52.6816379Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:32:52.6818520Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:32:52.6820563Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:32:52.6822587Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:32:52.6824484Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:32:52.6843821Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-06-05T08:32:53.7943219Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-06-05T08:32:54.1470884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-06-05T08:32:55.3158665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-06-05T08:32:55.5796833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-06-05T08:32:55.9207853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-06-05T08:32:56.9016571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-06-05T08:33:02.4011539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-06-05T08:33:02.7968974Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-06-05T08:33:02.8104588Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-06-05T08:33:02.8389132Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-06-05T08:33:02.8481845Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-06-05T08:33:02.8492162Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:02.8508273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-06-05T08:33:03.2166849Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-06-05T08:33:03.2299253Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-06-05T08:33:03.2631542Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-06-05T08:33:03.3422289Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-06-05T08:33:03.3539367Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-06-05T08:33:03.3826945Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50'
2024-06-05T08:33:03.4313941Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-06-05T08:33:03.4612778Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-06-05T08:33:03.4803884Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600'
2024-06-05T08:33:03.5660465Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-06-05T08:33:03.8400290Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781'
2024-06-05T08:33:03.8427832Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:03.8430233Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:03.8447569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2024-06-05T08:33:04.3005954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-06-05T08:33:05.3497839Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6'
2024-06-05T08:33:05.3768717Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242'
2024-06-05T08:33:05.4297710Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-06-05T08:33:05.4310757Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:05.4317040Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:05.4318822Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:05.4320610Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:05.4322804Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:05.4324996Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:05.4327218Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:05.4328992Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:05.4339698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-06-05T08:33:05.8812417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-06-05T08:33:06.9088963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-06-05T08:33:07.2172573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-06-05T08:33:13.3595806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-06-05T08:33:13.6344321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-06-05T08:33:13.8427898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-06-05T08:33:14.1521623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-06-05T08:33:20.5901452Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-06-05T08:33:20.6235787Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-06-05T08:33:20.6362847Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-06-05T08:33:20.7201227Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-06-05T08:33:20.7292226Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-06-05T08:33:20.7398763Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-06-05T08:33:20.7510396Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-06-05T08:33:20.7520776Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:20.7523173Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:20.7539800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-06-05T08:33:22.4984449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-06-05T08:33:23.6885336Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-06-05T08:33:23.7253151Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-06-05T08:33:24.0879234Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-06-05T08:33:24.0956370Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-06-05T08:33:24.3143353Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-06-05T08:33:24.3157628Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:24.3159437Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:24.3177100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-06-05T08:33:24.7078580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-06-05T08:33:25.7100455Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-06-05T08:33:25.7729549Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-06-05T08:33:25.7789095Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-06-05T08:33:25.7877410Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-06-05T08:33:25.8152820Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-06-05T08:33:25.8376543Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-06-05T08:33:25.8704033Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-06-05T08:33:25.8907085Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-06-05T08:33:25.8918071Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:25.8919801Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:25.8921650Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:25.8923743Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:25.8941192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-06-05T08:33:26.8450601Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-06-05T08:33:27.0826759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-06-05T08:33:28.0831784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-06-05T08:33:29.0285528Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-06-05T08:33:29.0400821Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-06-05T08:33:29.0964608Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-06-05T08:33:29.1193736Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-06-05T08:33:29.1204516Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:29.1221408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-06-05T08:33:29.3292331Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-06-05T08:33:29.3315109Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-06-05T08:33:29.3519026Z Entering 'android/libs/fbjni'
2024-06-05T08:33:29.3550785Z Entering 'third_party/FP16'
2024-06-05T08:33:29.3577780Z Entering 'third_party/FXdiv'
2024-06-05T08:33:29.3607136Z Entering 'third_party/NNPACK'
2024-06-05T08:33:29.3633488Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T08:33:29.3661665Z Entering 'third_party/XNNPACK'
2024-06-05T08:33:29.3700434Z Entering 'third_party/benchmark'
2024-06-05T08:33:29.3725804Z Entering 'third_party/cpp-httplib'
2024-06-05T08:33:29.3753478Z Entering 'third_party/cpuinfo'
2024-06-05T08:33:29.3781421Z Entering 'third_party/cudnn_frontend'
2024-06-05T08:33:29.3809301Z Entering 'third_party/cutlass'
2024-06-05T08:33:29.3840835Z Entering 'third_party/eigen'
2024-06-05T08:33:29.3869789Z Entering 'third_party/fbgemm'
2024-06-05T08:33:29.3896643Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:33:29.3921753Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:33:29.3950335Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:33:29.3981896Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:33:29.4009098Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:33:29.4038935Z Entering 'third_party/flatbuffers'
2024-06-05T08:33:29.4069619Z Entering 'third_party/fmt'
2024-06-05T08:33:29.4097513Z Entering 'third_party/foxi'
2024-06-05T08:33:29.4124191Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:33:29.4150489Z Entering 'third_party/gloo'
2024-06-05T08:33:29.4180623Z Entering 'third_party/googletest'
2024-06-05T08:33:29.4206987Z Entering 'third_party/ideep'
2024-06-05T08:33:29.4233893Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T08:33:29.4265474Z Entering 'third_party/ittapi'
2024-06-05T08:33:29.4294408Z Entering 'third_party/kineto'
2024-06-05T08:33:29.4321868Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:33:29.4350713Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:33:29.4377698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:33:29.4403811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:33:29.4431591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:33:29.4459421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:29.4488697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:33:29.4515575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:33:29.4543577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:33:29.4571781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:33:29.4599214Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:33:29.4625439Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:33:29.4653484Z Entering 'third_party/mimalloc'
2024-06-05T08:33:29.4682741Z Entering 'third_party/nccl/nccl'
2024-06-05T08:33:29.4712606Z Entering 'third_party/nlohmann'
2024-06-05T08:33:29.4740743Z Entering 'third_party/onnx'
2024-06-05T08:33:29.4781209Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:29.4810354Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:29.4838703Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T08:33:29.4869485Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:29.4897665Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:29.4923955Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:29.4950923Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:29.4978102Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:29.5004076Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:29.5031586Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:29.5059485Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:29.5086098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:29.5115574Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:29.5157196Z Entering 'third_party/pocketfft'
2024-06-05T08:33:29.5190315Z Entering 'third_party/protobuf'
2024-06-05T08:33:29.5221875Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:29.5247032Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:29.5274991Z Entering 'third_party/psimd'
2024-06-05T08:33:29.5301334Z Entering 'third_party/pthreadpool'
2024-06-05T08:33:29.5327712Z Entering 'third_party/pybind11'
2024-06-05T08:33:29.5355927Z Entering 'third_party/python-peachpy'
2024-06-05T08:33:29.5383054Z Entering 'third_party/sleef'
2024-06-05T08:33:29.5410736Z Entering 'third_party/tensorpipe'
2024-06-05T08:33:29.5438587Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:29.5464855Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:29.5492205Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:29.5519834Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:29.5544605Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:29.5584561Z ##[endgroup]
2024-06-05T08:33:29.5586506Z ##[group]Persisting credentials for submodules
2024-06-05T08:33:29.5589968Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-06-05T08:33:29.5799026Z Entering 'android/libs/fbjni'
2024-06-05T08:33:29.5834489Z Entering 'third_party/FP16'
2024-06-05T08:33:29.5870261Z Entering 'third_party/FXdiv'
2024-06-05T08:33:29.5904048Z Entering 'third_party/NNPACK'
2024-06-05T08:33:29.5940649Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T08:33:29.5974158Z Entering 'third_party/XNNPACK'
2024-06-05T08:33:29.6019341Z Entering 'third_party/benchmark'
2024-06-05T08:33:29.6055310Z Entering 'third_party/cpp-httplib'
2024-06-05T08:33:29.6090116Z Entering 'third_party/cpuinfo'
2024-06-05T08:33:29.6125588Z Entering 'third_party/cudnn_frontend'
2024-06-05T08:33:29.6159672Z Entering 'third_party/cutlass'
2024-06-05T08:33:29.6203396Z Entering 'third_party/eigen'
2024-06-05T08:33:29.6240099Z Entering 'third_party/fbgemm'
2024-06-05T08:33:29.6275670Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:33:29.6310547Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:33:29.6345127Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:33:29.6382285Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:33:29.6416509Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:33:29.6449750Z Entering 'third_party/flatbuffers'
2024-06-05T08:33:29.6487208Z Entering 'third_party/fmt'
2024-06-05T08:33:29.6520908Z Entering 'third_party/foxi'
2024-06-05T08:33:29.6554390Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:33:29.6586677Z Entering 'third_party/gloo'
2024-06-05T08:33:29.6621427Z Entering 'third_party/googletest'
2024-06-05T08:33:29.6657282Z Entering 'third_party/ideep'
2024-06-05T08:33:29.6693122Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T08:33:29.6733470Z Entering 'third_party/ittapi'
2024-06-05T08:33:29.6771481Z Entering 'third_party/kineto'
2024-06-05T08:33:29.6805650Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:33:29.6840737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:33:29.6876023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:33:29.6909890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:33:29.6943842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:33:29.6976405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:29.7012156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:33:29.7046028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:33:29.7080754Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:33:29.7116495Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:33:29.7152182Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:33:29.7184643Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:33:29.7220467Z Entering 'third_party/mimalloc'
2024-06-05T08:33:29.7255285Z Entering 'third_party/nccl/nccl'
2024-06-05T08:33:29.7291297Z Entering 'third_party/nlohmann'
2024-06-05T08:33:29.7327686Z Entering 'third_party/onnx'
2024-06-05T08:33:29.7373443Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:29.7412674Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:29.7450444Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T08:33:29.7484178Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:29.7517985Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:29.7553043Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:29.7585835Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:29.7620812Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:29.7653361Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:29.7687137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:29.7719990Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:29.7755772Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:29.7791741Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:29.7842425Z Entering 'third_party/pocketfft'
2024-06-05T08:33:29.7882605Z Entering 'third_party/protobuf'
2024-06-05T08:33:29.7919706Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:29.7953569Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:29.7991589Z Entering 'third_party/psimd'
2024-06-05T08:33:29.8025282Z Entering 'third_party/pthreadpool'
2024-06-05T08:33:29.8060019Z Entering 'third_party/pybind11'
2024-06-05T08:33:29.8092900Z Entering 'third_party/python-peachpy'
2024-06-05T08:33:29.8127556Z Entering 'third_party/sleef'
2024-06-05T08:33:29.8162112Z Entering 'third_party/tensorpipe'
2024-06-05T08:33:29.8196452Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:29.8229929Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:29.8261997Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:29.8295934Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:29.8329505Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:29.8376995Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-06-05T08:33:29.8578861Z Entering 'android/libs/fbjni'
2024-06-05T08:33:29.8609483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-06-05T08:33:29.8621929Z Entering 'third_party/FP16'
2024-06-05T08:33:29.8654320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-06-05T08:33:29.8667977Z Entering 'third_party/FXdiv'
2024-06-05T08:33:29.8699932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-06-05T08:33:29.8712674Z Entering 'third_party/NNPACK'
2024-06-05T08:33:29.8743783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-06-05T08:33:29.8757229Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T08:33:29.8789791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-06-05T08:33:29.8802448Z Entering 'third_party/XNNPACK'
2024-06-05T08:33:29.8834821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-06-05T08:33:29.8858400Z Entering 'third_party/benchmark'
2024-06-05T08:33:29.8890119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-06-05T08:33:29.8902365Z Entering 'third_party/cpp-httplib'
2024-06-05T08:33:29.8932892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-06-05T08:33:29.8946447Z Entering 'third_party/cpuinfo'
2024-06-05T08:33:29.8978595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-06-05T08:33:29.8992483Z Entering 'third_party/cudnn_frontend'
2024-06-05T08:33:29.9023295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-06-05T08:33:29.9037321Z Entering 'third_party/cutlass'
2024-06-05T08:33:29.9070665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-06-05T08:33:29.9089118Z Entering 'third_party/eigen'
2024-06-05T08:33:29.9120644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-06-05T08:33:29.9136773Z Entering 'third_party/fbgemm'
2024-06-05T08:33:29.9170178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-06-05T08:33:29.9182851Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:33:29.9214512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-06-05T08:33:29.9227710Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:33:29.9259412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-06-05T08:33:29.9271589Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:33:29.9304212Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-06-05T08:33:29.9320093Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:33:29.9353162Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:29.9366008Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:33:29.9397385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-06-05T08:33:29.9411261Z Entering 'third_party/flatbuffers'
2024-06-05T08:33:29.9443915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-06-05T08:33:29.9460011Z Entering 'third_party/fmt'
2024-06-05T08:33:29.9491869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-06-05T08:33:29.9505628Z Entering 'third_party/foxi'
2024-06-05T08:33:29.9536979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2024-06-05T08:33:29.9550882Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:33:29.9583134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-06-05T08:33:29.9594777Z Entering 'third_party/gloo'
2024-06-05T08:33:29.9626465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-06-05T08:33:29.9639620Z Entering 'third_party/googletest'
2024-06-05T08:33:29.9671669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:29.9684464Z Entering 'third_party/ideep'
2024-06-05T08:33:29.9716777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-06-05T08:33:29.9729769Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T08:33:29.9761343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-06-05T08:33:29.9778744Z Entering 'third_party/ittapi'
2024-06-05T08:33:29.9810573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-06-05T08:33:29.9823628Z Entering 'third_party/kineto'
2024-06-05T08:33:29.9854587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-06-05T08:33:29.9867705Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:33:29.9899877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-06-05T08:33:29.9911869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:33:29.9944555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-06-05T08:33:29.9958391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:33:29.9989600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-06-05T08:33:30.0001065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:33:30.0033418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-06-05T08:33:30.0046783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:33:30.0076700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-06-05T08:33:30.0087487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:30.0117789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-06-05T08:33:30.0132166Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:33:30.0162864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-06-05T08:33:30.0175401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:33:30.0207226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:30.0219538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:33:30.0251819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-06-05T08:33:30.0265475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:33:30.0296975Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-06-05T08:33:30.0309576Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:33:30.0340460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-06-05T08:33:30.0353535Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:33:30.0383983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:30.0397807Z Entering 'third_party/mimalloc'
2024-06-05T08:33:30.0432188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-06-05T08:33:30.0444250Z Entering 'third_party/nccl/nccl'
2024-06-05T08:33:30.0476316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-06-05T08:33:30.0488305Z Entering 'third_party/nlohmann'
2024-06-05T08:33:30.0519008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-06-05T08:33:30.0533839Z Entering 'third_party/onnx'
2024-06-05T08:33:30.0565311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-06-05T08:33:30.0591591Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:30.0622957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2024-06-05T08:33:30.0635285Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:30.0668084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-06-05T08:33:30.0683250Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T08:33:30.0717500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-06-05T08:33:30.0733430Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:30.0765443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-06-05T08:33:30.0778035Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:30.0809603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:30.0823294Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:30.0854764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-06-05T08:33:30.0867552Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:30.0898754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-06-05T08:33:30.0912290Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:30.0944494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-06-05T08:33:30.0956734Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:30.0989385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-06-05T08:33:30.1001068Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:30.1034353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-06-05T08:33:30.1046232Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:30.1077687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-06-05T08:33:30.1091603Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:30.1121539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-06-05T08:33:30.1136195Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:30.1165932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-06-05T08:33:30.1195070Z Entering 'third_party/pocketfft'
2024-06-05T08:33:30.1227018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-06-05T08:33:30.1240715Z Entering 'third_party/protobuf'
2024-06-05T08:33:30.1270969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-06-05T08:33:30.1284166Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:30.1316490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-06-05T08:33:30.1329472Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:30.1361475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:30.1376868Z Entering 'third_party/psimd'
2024-06-05T08:33:30.1409137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-06-05T08:33:30.1421975Z Entering 'third_party/pthreadpool'
2024-06-05T08:33:30.1452871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-06-05T08:33:30.1465092Z Entering 'third_party/pybind11'
2024-06-05T08:33:30.1494598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-06-05T08:33:30.1506880Z Entering 'third_party/python-peachpy'
2024-06-05T08:33:30.1537064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-06-05T08:33:30.1547979Z Entering 'third_party/sleef'
2024-06-05T08:33:30.1579745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-06-05T08:33:30.1593248Z Entering 'third_party/tensorpipe'
2024-06-05T08:33:30.1625169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-06-05T08:33:30.1637693Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:30.1669107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-06-05T08:33:30.1681676Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:30.1713250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-06-05T08:33:30.1725676Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:30.1758225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-06-05T08:33:30.1771675Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:30.1803295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-06-05T08:33:30.1813642Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:30.1845501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-06-05T08:33:30.2630112Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-06-05T08:33:30.2829194Z Entering 'android/libs/fbjni'
2024-06-05T08:33:30.2854559Z Entering 'third_party/FP16'
2024-06-05T08:33:30.2880503Z Entering 'third_party/FXdiv'
2024-06-05T08:33:30.2909475Z Entering 'third_party/NNPACK'
2024-06-05T08:33:30.2938247Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T08:33:30.2965048Z Entering 'third_party/XNNPACK'
2024-06-05T08:33:30.3002327Z Entering 'third_party/benchmark'
2024-06-05T08:33:30.3030795Z Entering 'third_party/cpp-httplib'
2024-06-05T08:33:30.3059664Z Entering 'third_party/cpuinfo'
2024-06-05T08:33:30.3085765Z Entering 'third_party/cudnn_frontend'
2024-06-05T08:33:30.3114044Z Entering 'third_party/cutlass'
2024-06-05T08:33:30.3144813Z Entering 'third_party/eigen'
2024-06-05T08:33:30.3174414Z Entering 'third_party/fbgemm'
2024-06-05T08:33:30.3201141Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:33:30.3228973Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:33:30.3257232Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:33:30.3289403Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:33:30.3316996Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:33:30.3344347Z Entering 'third_party/flatbuffers'
2024-06-05T08:33:30.3375792Z Entering 'third_party/fmt'
2024-06-05T08:33:30.3404996Z Entering 'third_party/foxi'
2024-06-05T08:33:30.3434710Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:33:30.3463972Z Entering 'third_party/gloo'
2024-06-05T08:33:30.3494236Z Entering 'third_party/googletest'
2024-06-05T08:33:30.3523899Z Entering 'third_party/ideep'
2024-06-05T08:33:30.3551278Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T08:33:30.3584661Z Entering 'third_party/ittapi'
2024-06-05T08:33:30.3614295Z Entering 'third_party/kineto'
2024-06-05T08:33:30.3641183Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:33:30.3669928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:33:30.3698364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:33:30.3724070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:33:30.3752629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:33:30.3779835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:30.3808380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:33:30.3835535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:33:30.3863733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:33:30.3889966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:33:30.3916458Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:33:30.3945576Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:33:30.3974890Z Entering 'third_party/mimalloc'
2024-06-05T08:33:30.4002972Z Entering 'third_party/nccl/nccl'
2024-06-05T08:33:30.4032220Z Entering 'third_party/nlohmann'
2024-06-05T08:33:30.4060140Z Entering 'third_party/onnx'
2024-06-05T08:33:30.4098464Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:30.4127322Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:30.4156165Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T08:33:30.4187175Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:30.4215042Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:30.4242390Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:30.4270134Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:30.4330924Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:30.4359281Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:30.4386074Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:30.4410937Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:30.4440412Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:30.4468364Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:30.4511364Z Entering 'third_party/pocketfft'
2024-06-05T08:33:30.4540828Z Entering 'third_party/protobuf'
2024-06-05T08:33:30.4570711Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:30.4598351Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:30.4629146Z Entering 'third_party/psimd'
2024-06-05T08:33:30.4655943Z Entering 'third_party/pthreadpool'
2024-06-05T08:33:30.4683660Z Entering 'third_party/pybind11'
2024-06-05T08:33:30.4712630Z Entering 'third_party/python-peachpy'
2024-06-05T08:33:30.4739163Z Entering 'third_party/sleef'
2024-06-05T08:33:30.4767902Z Entering 'third_party/tensorpipe'
2024-06-05T08:33:30.4795550Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:30.4821977Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:30.4847711Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:30.4874773Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:30.4902010Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:30.4940012Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-06-05T08:33:30.5146316Z Entering 'android/libs/fbjni'
2024-06-05T08:33:30.5176202Z Entering 'third_party/FP16'
2024-06-05T08:33:30.5206209Z Entering 'third_party/FXdiv'
2024-06-05T08:33:30.5233091Z Entering 'third_party/NNPACK'
2024-06-05T08:33:30.5260529Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T08:33:30.5287010Z Entering 'third_party/XNNPACK'
2024-06-05T08:33:30.5326864Z Entering 'third_party/benchmark'
2024-06-05T08:33:30.5355912Z Entering 'third_party/cpp-httplib'
2024-06-05T08:33:30.5383659Z Entering 'third_party/cpuinfo'
2024-06-05T08:33:30.5410634Z Entering 'third_party/cudnn_frontend'
2024-06-05T08:33:30.5438582Z Entering 'third_party/cutlass'
2024-06-05T08:33:30.5470174Z Entering 'third_party/eigen'
2024-06-05T08:33:30.5497733Z Entering 'third_party/fbgemm'
2024-06-05T08:33:30.5525535Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T08:33:30.5553312Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T08:33:30.5581479Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T08:33:30.5611550Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T08:33:30.5637743Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T08:33:30.5666819Z Entering 'third_party/flatbuffers'
2024-06-05T08:33:30.5698573Z Entering 'third_party/fmt'
2024-06-05T08:33:30.5728765Z Entering 'third_party/foxi'
2024-06-05T08:33:30.5756340Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T08:33:30.5786782Z Entering 'third_party/gloo'
2024-06-05T08:33:30.5811853Z Entering 'third_party/googletest'
2024-06-05T08:33:30.5838295Z Entering 'third_party/ideep'
2024-06-05T08:33:30.5866471Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T08:33:30.5898238Z Entering 'third_party/ittapi'
2024-06-05T08:33:30.5924621Z Entering 'third_party/kineto'
2024-06-05T08:33:30.5952577Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T08:33:30.5982280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T08:33:30.6012199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T08:33:30.6040418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T08:33:30.6066942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T08:33:30.6095136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T08:33:30.6122329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T08:33:30.6151053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T08:33:30.6179883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T08:33:30.6209116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T08:33:30.6237319Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T08:33:30.6265668Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T08:33:30.6294556Z Entering 'third_party/mimalloc'
2024-06-05T08:33:30.6320809Z Entering 'third_party/nccl/nccl'
2024-06-05T08:33:30.6350317Z Entering 'third_party/nlohmann'
2024-06-05T08:33:30.6379072Z Entering 'third_party/onnx'
2024-06-05T08:33:30.6419304Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T08:33:30.6448231Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T08:33:30.6478056Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T08:33:30.6507121Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T08:33:30.6535319Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T08:33:30.6562915Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T08:33:30.6590823Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T08:33:30.6620435Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T08:33:30.6649376Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T08:33:30.6677681Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T08:33:30.6705388Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T08:33:30.6733558Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T08:33:30.6762864Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T08:33:30.6804840Z Entering 'third_party/pocketfft'
2024-06-05T08:33:30.6834535Z Entering 'third_party/protobuf'
2024-06-05T08:33:30.6863365Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T08:33:30.6890621Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T08:33:30.6917101Z Entering 'third_party/psimd'
2024-06-05T08:33:30.6946016Z Entering 'third_party/pthreadpool'
2024-06-05T08:33:30.6975855Z Entering 'third_party/pybind11'
2024-06-05T08:33:30.7003134Z Entering 'third_party/python-peachpy'
2024-06-05T08:33:30.7032463Z Entering 'third_party/sleef'
2024-06-05T08:33:30.7059938Z Entering 'third_party/tensorpipe'
2024-06-05T08:33:30.7087530Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T08:33:30.7115104Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T08:33:30.7141977Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T08:33:30.7170019Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T08:33:30.7197545Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T08:33:30.7237034Z ##[endgroup]
2024-06-05T08:33:30.7266110Z [command]/usr/bin/git log -1 --format='%H'
2024-06-05T08:33:30.7287451Z 'dffed71f3397e435f3656f25960a4d75ad415746'
2024-06-05T08:33:30.7462583Z Prepare all required actions
2024-06-05T08:33:30.7463075Z Getting action download info
2024-06-05T08:33:30.8960883Z ##[group]Run ./.github/actions/setup-linux
2024-06-05T08:33:30.8961347Z env:
2024-06-05T08:33:30.8961671Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:30.8962158Z ##[endgroup]
2024-06-05T08:33:30.9021821Z ##[group]Run set -euo pipefail
2024-06-05T08:33:30.9022293Z set -euo pipefail
2024-06-05T08:33:30.9022723Z function get_ec2_metadata() {
2024-06-05T08:33:30.9023300Z   # Pulled from instance metadata endpoint for EC2
2024-06-05T08:33:30.9024240Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-06-05T08:33:30.9025052Z   category=$1
2024-06-05T08:33:30.9025616Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-06-05T08:33:30.9026301Z   runner_name_str=i-07bc38970f73de3fc
2024-06-05T08:33:30.9026830Z   if [[ -f /.inarc ]]; then
2024-06-05T08:33:30.9027358Z     echo "ARC Runner, no info on ec2 metadata"
2024-06-05T08:33:30.9027966Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-06-05T08:33:30.9028700Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-06-05T08:33:30.9029346Z   else
2024-06-05T08:33:30.9029870Z     curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-06-05T08:33:30.9030475Z   fi
2024-06-05T08:33:30.9030789Z }
2024-06-05T08:33:30.9031181Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-06-05T08:33:30.9031797Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-06-05T08:33:30.9032496Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-06-05T08:33:30.9033102Z echo "system info $(uname -a)"
2024-06-05T08:33:30.9041030Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:30.9041554Z env:
2024-06-05T08:33:30.9041864Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:30.9042345Z ##[endgroup]
2024-06-05T08:33:30.9111150Z ami-id: ami-0ce0c36d7a00b20e2
2024-06-05T08:33:30.9153565Z instance-id: i-07bc38970f73de3fc
2024-06-05T08:33:30.9194064Z instance-type: g5.4xlarge
2024-06-05T08:33:30.9200203Z system info Linux ip-10-0-62-168.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-06-05T08:33:30.9221031Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-06-05T08:33:30.9222006Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-06-05T08:33:30.9229905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:30.9230449Z env:
2024-06-05T08:33:30.9230774Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:30.9231165Z ##[endgroup]
2024-06-05T08:33:30.9294583Z ##[group]Run if systemctl is-active --quiet docker; then
2024-06-05T08:33:30.9295247Z if systemctl is-active --quiet docker; then
2024-06-05T08:33:30.9295840Z     echo "Docker daemon is running...";
2024-06-05T08:33:30.9296363Z else
2024-06-05T08:33:30.9296927Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-06-05T08:33:30.9297576Z fi
2024-06-05T08:33:30.9305078Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:30.9305624Z env:
2024-06-05T08:33:30.9305933Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:30.9306320Z ##[endgroup]
2024-06-05T08:33:30.9341672Z Docker daemon is running...
2024-06-05T08:33:30.9387179Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-06-05T08:33:30.9387772Z with:
2024-06-05T08:33:30.9388067Z   shell: bash
2024-06-05T08:33:30.9388389Z   timeout_minutes: 5
2024-06-05T08:33:30.9388746Z   max_attempts: 3
2024-06-05T08:33:30.9389093Z   retry_wait_seconds: 30
2024-06-05T08:33:30.9390651Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

2024-06-05T08:33:30.9392143Z   polling_interval_seconds: 1
2024-06-05T08:33:30.9392566Z   warning_on_retry: true
2024-06-05T08:33:30.9392954Z   continue_on_error: false
2024-06-05T08:33:30.9393323Z env:
2024-06-05T08:33:30.9393618Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:30.9394011Z   AWS_RETRY_MODE: standard
2024-06-05T08:33:30.9394395Z   AWS_MAX_ATTEMPTS: 5
2024-06-05T08:33:30.9394765Z   AWS_DEFAULT_REGION: us-east-1
2024-06-05T08:33:30.9395169Z ##[endgroup]
2024-06-05T08:33:31.8039917Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-06-05T08:33:31.8040818Z Configure a credential helper to remove this warning. See
2024-06-05T08:33:31.8041801Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-06-05T08:33:31.8042731Z 
2024-06-05T08:33:31.8042953Z Login Succeeded
2024-06-05T08:33:31.9886731Z Command completed after 1 attempt(s).
2024-06-05T08:33:31.9928103Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-06-05T08:33:31.9928868Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-06-05T08:33:31.9929581Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-06-05T08:33:31.9937406Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:31.9937938Z env:
2024-06-05T08:33:31.9938251Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:31.9938633Z ##[endgroup]
2024-06-05T08:33:31.9998145Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-06-05T08:33:31.9998976Z # ignore expansion of "docker ps -q" since it could be empty
2024-06-05T08:33:31.9999622Z # shellcheck disable=SC2046
2024-06-05T08:33:32.0000125Z docker stop $(docker ps -q) || true
2024-06-05T08:33:32.0000655Z # Prune all of the docker images
2024-06-05T08:33:32.0001146Z docker system prune -af
2024-06-05T08:33:32.0008666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.0009211Z env:
2024-06-05T08:33:32.0009530Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.0010033Z ##[endgroup]
2024-06-05T08:33:32.0311328Z "docker stop" requires at least 1 argument.
2024-06-05T08:33:32.0312061Z See 'docker stop --help'.
2024-06-05T08:33:32.0312322Z 
2024-06-05T08:33:32.0312568Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-06-05T08:33:32.0312987Z 
2024-06-05T08:33:32.0313279Z Stop one or more running containers
2024-06-05T08:33:32.0488727Z Total reclaimed space: 0B
2024-06-05T08:33:32.0522982Z ##[group]Run set +e
2024-06-05T08:33:32.0523365Z set +e
2024-06-05T08:33:32.0523696Z set -x
2024-06-05T08:33:32.0524027Z 
2024-06-05T08:33:32.0524376Z PT_DOMAIN=download.pytorch.org
2024-06-05T08:33:32.0525226Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-06-05T08:33:32.0526367Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-06-05T08:33:32.0527484Z # one is returned at random
2024-06-05T08:33:32.0528090Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-06-05T08:33:32.0528638Z 
2024-06-05T08:33:32.0528992Z if [ -z "${RESOLVED_IP}" ]; then
2024-06-05T08:33:32.0529660Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-06-05T08:33:32.0530478Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-06-05T08:33:32.0531070Z 
2024-06-05T08:33:32.0531428Z   if [ -z "${RESOLVED_IP}" ]; then
2024-06-05T08:33:32.0532024Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-06-05T08:33:32.0532570Z     exit 1
2024-06-05T08:33:32.0532908Z   fi
2024-06-05T08:33:32.0533228Z fi
2024-06-05T08:33:32.0533682Z 
2024-06-05T08:33:32.0534084Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-06-05T08:33:32.0534638Z   # Clean up any old records first
2024-06-05T08:33:32.0535196Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-06-05T08:33:32.0535691Z fi
2024-06-05T08:33:32.0535989Z 
2024-06-05T08:33:32.0536457Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-06-05T08:33:32.0537089Z cat /etc/hosts
2024-06-05T08:33:32.0544821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.0545347Z env:
2024-06-05T08:33:32.0545653Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.0546041Z ##[endgroup]
2024-06-05T08:33:32.0569181Z + PT_DOMAIN=download.pytorch.org
2024-06-05T08:33:32.0573136Z ++ dig -4 +short download.pytorch.org
2024-06-05T08:33:32.0573744Z ++ tail -n1
2024-06-05T08:33:32.0669099Z + RESOLVED_IP=18.160.10.76
2024-06-05T08:33:32.0669574Z + '[' -z 18.160.10.76 ']'
2024-06-05T08:33:32.0670048Z + grep -r download.pytorch.org /etc/hosts
2024-06-05T08:33:32.0674706Z 18.160.10.28 download.pytorch.org
2024-06-05T08:33:32.0675510Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2024-06-05T08:33:32.0770672Z + echo '18.160.10.76 download.pytorch.org'
2024-06-05T08:33:32.0771214Z + sudo tee -a /etc/hosts
2024-06-05T08:33:32.1038958Z 18.160.10.76 download.pytorch.org
2024-06-05T08:33:32.1052555Z + cat /etc/hosts
2024-06-05T08:33:32.1057395Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-06-05T08:33:32.1069063Z ::1         localhost6 localhost6.localdomain6
2024-06-05T08:33:32.1069603Z 18.160.10.76 download.pytorch.org
2024-06-05T08:33:32.1268929Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2024-06-05T08:33:32.1269584Z with:
2024-06-05T08:33:32.1270751Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1272046Z   docker-build-dir: .ci/docker
2024-06-05T08:33:32.1272461Z   working-directory: .
2024-06-05T08:33:32.1272965Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.1273656Z   force-push: false
2024-06-05T08:33:32.1273981Z env:
2024-06-05T08:33:32.1274283Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.1274664Z ##[endgroup]
2024-06-05T08:33:32.1293515Z ##[group]Run set -ex
2024-06-05T08:33:32.1293927Z set -ex
2024-06-05T08:33:32.1294244Z 
2024-06-05T08:33:32.1294846Z # If the docker build directory or the build script doesn't exist, the action will
2024-06-05T08:33:32.1295903Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-06-05T08:33:32.1296804Z # job could then download the pre-built image as usual
2024-06-05T08:33:32.1297598Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-06-05T08:33:32.1298315Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1298998Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1299597Z 
2024-06-05T08:33:32.1300145Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-06-05T08:33:32.1300818Z   exit 0
2024-06-05T08:33:32.1301159Z else
2024-06-05T08:33:32.1301566Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1302046Z fi
2024-06-05T08:33:32.1302353Z 
2024-06-05T08:33:32.1302862Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-06-05T08:33:32.1303761Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-06-05T08:33:32.1304580Z   # use it as it is, but first let's extract the tag
2024-06-05T08:33:32.1305330Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-06-05T08:33:32.1306099Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1306849Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1307438Z else
2024-06-05T08:33:32.1307915Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-06-05T08:33:32.1308626Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1309577Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1310362Z fi
2024-06-05T08:33:32.1318029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.1318565Z env:
2024-06-05T08:33:32.1318875Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.1319257Z   REPO_NAME: pytorch
2024-06-05T08:33:32.1320439Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1321715Z   DOCKER_BUILD_DIR: .ci/docker
2024-06-05T08:33:32.1322337Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.1322895Z ##[endgroup]
2024-06-05T08:33:32.1344763Z + [[ ! -d .ci/docker ]]
2024-06-05T08:33:32.1345197Z + [[ ! -f .ci/docker/build.sh ]]
2024-06-05T08:33:32.1345607Z + echo skip=false
2024-06-05T08:33:32.1347645Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-06-05T08:33:32.1350516Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1351769Z ++ awk -F '[:,]' '{print $2}'
2024-06-05T08:33:32.1356598Z + DOCKER_TAG=28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1357351Z + echo docker-tag=28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1359121Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1386473Z ##[group]Run set +e
2024-06-05T08:33:32.1386876Z set +e
2024-06-05T08:33:32.1387206Z set -x
2024-06-05T08:33:32.1387521Z 
2024-06-05T08:33:32.1387831Z login() {
2024-06-05T08:33:32.1388551Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-06-05T08:33:32.1389313Z }
2024-06-05T08:33:32.1389609Z 
2024-06-05T08:33:32.1389918Z retry () {
2024-06-05T08:33:32.1390340Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-06-05T08:33:32.1390817Z }
2024-06-05T08:33:32.1391113Z 
2024-06-05T08:33:32.1391459Z retry login "${DOCKER_REGISTRY}"
2024-06-05T08:33:32.1391908Z 
2024-06-05T08:33:32.1392410Z # Check if image already exists, if it does then skip building it
2024-06-05T08:33:32.1393159Z if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-06-05T08:33:32.1393716Z   exit 0
2024-06-05T08:33:32.1394045Z fi
2024-06-05T08:33:32.1394344Z 
2024-06-05T08:33:32.1394880Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-06-05T08:33:32.1395759Z # be empty.  The default action would be to continue rebuild the image
2024-06-05T08:33:32.1396556Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-06-05T08:33:32.1397271Z   # if we're on the base branch then use the parent commit
2024-06-05T08:33:32.1397907Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-06-05T08:33:32.1398375Z else
2024-06-05T08:33:32.1398885Z   # otherwise we're on a PR, so use the most recent base commit
2024-06-05T08:33:32.1399623Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-06-05T08:33:32.1400168Z fi
2024-06-05T08:33:32.1400472Z 
2024-06-05T08:33:32.1400814Z if [[ -z "${MERGE_BASE}" ]]; then
2024-06-05T08:33:32.1401356Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1401850Z 
2024-06-05T08:33:32.1402642Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-06-05T08:33:32.1403450Z   exit 0
2024-06-05T08:33:32.1403771Z fi
2024-06-05T08:33:32.1404078Z 
2024-06-05T08:33:32.1404551Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-06-05T08:33:32.1405575Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-06-05T08:33:32.1406564Z   exit 1
2024-06-05T08:33:32.1406955Z fi
2024-06-05T08:33:32.1407263Z 
2024-06-05T08:33:32.1407791Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-06-05T08:33:32.1408796Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-06-05T08:33:32.1409729Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-06-05T08:33:32.1410758Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-06-05T08:33:32.1411927Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-06-05T08:33:32.1412738Z fi
2024-06-05T08:33:32.1413048Z 
2024-06-05T08:33:32.1413441Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-06-05T08:33:32.1420387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.1420921Z env:
2024-06-05T08:33:32.1421222Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.1421621Z   DOCKER_BUILD_DIR: .ci/docker
2024-06-05T08:33:32.1422123Z   BASE_REVISION: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:33:32.1423447Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1424849Z   DOCKER_TAG: 28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.1425486Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.1426032Z ##[endgroup]
2024-06-05T08:33:32.1446384Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.1447506Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.1448131Z + aws ecr get-login-password --region us-east-1
2024-06-05T08:33:32.1448945Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.5570564Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-06-05T08:33:32.5571531Z Configure a credential helper to remove this warning. See
2024-06-05T08:33:32.5572537Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-06-05T08:33:32.5573120Z 
2024-06-05T08:33:32.5573270Z Login Succeeded
2024-06-05T08:33:32.5584851Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.7605138Z {
2024-06-05T08:33:32.7605565Z 	"schemaVersion": 2,
2024-06-05T08:33:32.7606358Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-06-05T08:33:32.7607352Z 	"config": {
2024-06-05T08:33:32.7608022Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-06-05T08:33:32.7608781Z 		"size": 45560,
2024-06-05T08:33:32.7609359Z 		"digest": "sha256:4fa961484e1008b77cf438ac442612692dbee4f3476a753574ace973e65df56a"
2024-06-05T08:33:32.7610018Z 	},
2024-06-05T08:33:32.7610297Z 	"layers": [
2024-06-05T08:33:32.7610691Z 		{
2024-06-05T08:33:32.7611352Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7612171Z 			"size": 28584317,
2024-06-05T08:33:32.7612857Z 			"digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05"
2024-06-05T08:33:32.7613555Z 		},
2024-06-05T08:33:32.7613962Z 		{
2024-06-05T08:33:32.7614624Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7615436Z 			"size": 7944771,
2024-06-05T08:33:32.7616088Z 			"digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6"
2024-06-05T08:33:32.7616757Z 		},
2024-06-05T08:33:32.7617030Z 		{
2024-06-05T08:33:32.7617492Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7618083Z 			"size": 57593718,
2024-06-05T08:33:32.7618663Z 			"digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481"
2024-06-05T08:33:32.7619299Z 		},
2024-06-05T08:33:32.7619573Z 		{
2024-06-05T08:33:32.7620035Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7620630Z 			"size": 187,
2024-06-05T08:33:32.7621364Z 			"digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6"
2024-06-05T08:33:32.7622302Z 		},
2024-06-05T08:33:32.7622955Z 		{
2024-06-05T08:33:32.7623363Z + exit 0
2024-06-05T08:33:32.7624002Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7624688Z 			"size": 6885,
2024-06-05T08:33:32.7625464Z 			"digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488"
2024-06-05T08:33:32.7626571Z 		},
2024-06-05T08:33:32.7626953Z 		{
2024-06-05T08:33:32.7627455Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7628047Z 			"size": 1361380219,
2024-06-05T08:33:32.7628638Z 			"digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46"
2024-06-05T08:33:32.7629276Z 		},
2024-06-05T08:33:32.7629550Z 		{
2024-06-05T08:33:32.7630069Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7630704Z 			"size": 62686,
2024-06-05T08:33:32.7631294Z 			"digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3"
2024-06-05T08:33:32.7632137Z 		},
2024-06-05T08:33:32.7632432Z 		{
2024-06-05T08:33:32.7632894Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7633662Z 			"size": 1685,
2024-06-05T08:33:32.7634450Z 			"digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9"
2024-06-05T08:33:32.7635301Z 		},
2024-06-05T08:33:32.7635839Z 		{
2024-06-05T08:33:32.7636462Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7637259Z 			"size": 1523,
2024-06-05T08:33:32.7638053Z 			"digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d"
2024-06-05T08:33:32.7638758Z 		},
2024-06-05T08:33:32.7639036Z 		{
2024-06-05T08:33:32.7639535Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7640146Z 			"size": 2608186849,
2024-06-05T08:33:32.7640747Z 			"digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8"
2024-06-05T08:33:32.7641402Z 		},
2024-06-05T08:33:32.7641665Z 		{
2024-06-05T08:33:32.7642296Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7642885Z 			"size": 86631,
2024-06-05T08:33:32.7643466Z 			"digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab"
2024-06-05T08:33:32.7644102Z 		},
2024-06-05T08:33:32.7644370Z 		{
2024-06-05T08:33:32.7644836Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7645413Z 			"size": 1822,
2024-06-05T08:33:32.7645968Z 			"digest": "sha256:70afc194f3617bc04d42896498cf82eb3c366f9a2131e81731c5517475cbb580"
2024-06-05T08:33:32.7646850Z 		},
2024-06-05T08:33:32.7647133Z 		{
2024-06-05T08:33:32.7647589Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7648179Z 			"size": 245767952,
2024-06-05T08:33:32.7648773Z 			"digest": "sha256:a15b9861c14e19f8a633a2fadc9599e35de77cf526a3833c437e41f66d3ff6e8"
2024-06-05T08:33:32.7649425Z 		},
2024-06-05T08:33:32.7649688Z 		{
2024-06-05T08:33:32.7650153Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7650738Z 			"size": 545,
2024-06-05T08:33:32.7651318Z 			"digest": "sha256:bcc9c2b688f4b2ccfdc60b121d3042d360a63b1d0e40b0d3b137fec7882eba21"
2024-06-05T08:33:32.7651962Z 		},
2024-06-05T08:33:32.7652234Z 		{
2024-06-05T08:33:32.7652699Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7653281Z 			"size": 1280,
2024-06-05T08:33:32.7653859Z 			"digest": "sha256:9bdedd1ac09a5f44d3224dced928736af9963d09471fdfe6426132e227e4e4a1"
2024-06-05T08:33:32.7654511Z 		},
2024-06-05T08:33:32.7654783Z 		{
2024-06-05T08:33:32.7655240Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7655819Z 			"size": 483,
2024-06-05T08:33:32.7656384Z 			"digest": "sha256:0d1cc19d82a491a2f7b0d495b52845d1df01234ac010848c1d38d0853bf63948"
2024-06-05T08:33:32.7657032Z 		},
2024-06-05T08:33:32.7657307Z 		{
2024-06-05T08:33:32.7657767Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7658353Z 			"size": 91711149,
2024-06-05T08:33:32.7658933Z 			"digest": "sha256:d5495d103885df797b93c17b6a592a7156fbd97d637b75347cb0aa73ba3bc040"
2024-06-05T08:33:32.7659562Z 		},
2024-06-05T08:33:32.7659831Z 		{
2024-06-05T08:33:32.7660291Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7660986Z 			"size": 3241,
2024-06-05T08:33:32.7661566Z 			"digest": "sha256:ea09054a5a20d2a5cd9f06725b51ec602a8c782971cbf3cbaf57941469bc09b3"
2024-06-05T08:33:32.7662210Z 		},
2024-06-05T08:33:32.7662480Z 		{
2024-06-05T08:33:32.7662934Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7663518Z 			"size": 1905,
2024-06-05T08:33:32.7664090Z 			"digest": "sha256:fbcdf69e3d41f18d50b9f7917ce461fa788604013e415585e095e67dba32b970"
2024-06-05T08:33:32.7664734Z 		},
2024-06-05T08:33:32.7664995Z 		{
2024-06-05T08:33:32.7665561Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7666141Z 			"size": 701,
2024-06-05T08:33:32.7666710Z 			"digest": "sha256:0b14b049729cab9bdb836a987d36f5166f3308c620eb4bb48c7f6a98170bb6ef"
2024-06-05T08:33:32.7667394Z 		},
2024-06-05T08:33:32.7667666Z 		{
2024-06-05T08:33:32.7668129Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7668704Z 			"size": 2798087078,
2024-06-05T08:33:32.7669296Z 			"digest": "sha256:465d7dda2107858ff42957a641da53eb6dcd8681627c603efbb711ff830491ac"
2024-06-05T08:33:32.7669940Z 		},
2024-06-05T08:33:32.7670210Z 		{
2024-06-05T08:33:32.7670663Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7671243Z 			"size": 379,
2024-06-05T08:33:32.7671798Z 			"digest": "sha256:950fe48ec9902701b010654544820ee8aa0fe4185c1ffae0d44e4e1753285f19"
2024-06-05T08:33:32.7672428Z 		},
2024-06-05T08:33:32.7672690Z 		{
2024-06-05T08:33:32.7673151Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7673740Z 			"size": 11892,
2024-06-05T08:33:32.7674319Z 			"digest": "sha256:4d2f5b21f13f792aae9d3afd21dc0e74e56b0e79bebfbec66c80f3cb375991ce"
2024-06-05T08:33:32.7674969Z 		},
2024-06-05T08:33:32.7675238Z 		{
2024-06-05T08:33:32.7675693Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7676269Z 			"size": 803,
2024-06-05T08:33:32.7676844Z 			"digest": "sha256:5e6df7aeac962725259b5424f0ae5193a101846ecc2e1abf9a4aab71986b5e10"
2024-06-05T08:33:32.7677488Z 		},
2024-06-05T08:33:32.7677758Z 		{
2024-06-05T08:33:32.7678210Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7678789Z 			"size": 106,
2024-06-05T08:33:32.7679346Z 			"digest": "sha256:76150770c77cc987344b35f78e6bc0e3ce5292ec8d96942eea7046c15cfb9e33"
2024-06-05T08:33:32.7679980Z 		},
2024-06-05T08:33:32.7680244Z 		{
2024-06-05T08:33:32.7680704Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7681290Z 			"size": 503,
2024-06-05T08:33:32.7681837Z 			"digest": "sha256:d076da148a138391f6073e3e1d7ca456766e82941376765b8053c8e7fd86b6cc"
2024-06-05T08:33:32.7682631Z 		},
2024-06-05T08:33:32.7682901Z 		{
2024-06-05T08:33:32.7683361Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7683941Z 			"size": 121477226,
2024-06-05T08:33:32.7684529Z 			"digest": "sha256:8517857fd1eeed245a3446f03634611ca9c6757e25ee4e2d001f9e3a0974b194"
2024-06-05T08:33:32.7685161Z 		},
2024-06-05T08:33:32.7685429Z 		{
2024-06-05T08:33:32.7685884Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7686639Z 			"size": 109,
2024-06-05T08:33:32.7687242Z 			"digest": "sha256:7e68d7c69ce8d5c22fa0c8df6347ea183345f4cd8bd294b4db4dd22471ed3317"
2024-06-05T08:33:32.7687904Z 		},
2024-06-05T08:33:32.7688168Z 		{
2024-06-05T08:33:32.7688629Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7689220Z 			"size": 490,
2024-06-05T08:33:32.7689793Z 			"digest": "sha256:c3ad3f0969a218726ef29d25dd6664b02091ad27e3cf4abaa743e2719ed8a867"
2024-06-05T08:33:32.7690447Z 		},
2024-06-05T08:33:32.7690716Z 		{
2024-06-05T08:33:32.7691177Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7691755Z 			"size": 298,
2024-06-05T08:33:32.7692327Z 			"digest": "sha256:4b05a1fe871da3c69cad4595775fb21f952bc5f7233882ce4c1922c1d8fe9942"
2024-06-05T08:33:32.7693076Z 		},
2024-06-05T08:33:32.7693352Z 		{
2024-06-05T08:33:32.7693810Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7694403Z 			"size": 103,
2024-06-05T08:33:32.7694981Z 			"digest": "sha256:7f2ff6e7943b8f918e6b38dae3eafe7625c6a8845d0cb251bf0f912fd1a22c1e"
2024-06-05T08:33:32.7695630Z 		},
2024-06-05T08:33:32.7695898Z 		{
2024-06-05T08:33:32.7696360Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7696974Z 			"size": 1473,
2024-06-05T08:33:32.7697570Z 			"digest": "sha256:fcf3589d3f0decc664433f34c399dfa0042b0a0f034b7fc7b78b67f5dd9858fe"
2024-06-05T08:33:32.7698323Z 		},
2024-06-05T08:33:32.7698598Z 		{
2024-06-05T08:33:32.7699059Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7699646Z 			"size": 446214722,
2024-06-05T08:33:32.7700254Z 			"digest": "sha256:bb7540904742bb8b963edda37f52aad5e478f26f3a8abf62fcfdb7fe9f2bfa8e"
2024-06-05T08:33:32.7700923Z 		},
2024-06-05T08:33:32.7701192Z 		{
2024-06-05T08:33:32.7701655Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7702241Z 			"size": 160,
2024-06-05T08:33:32.7702801Z 			"digest": "sha256:b1419f097390321e462ca82bae26197d3694ac807f9d09d7f3e83694ad9a22a8"
2024-06-05T08:33:32.7703430Z 		},
2024-06-05T08:33:32.7703702Z 		{
2024-06-05T08:33:32.7704161Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7704759Z 			"size": 564,
2024-06-05T08:33:32.7705334Z 			"digest": "sha256:fe30e7b83e01ea003ca4939eb4c368ede37b66be2171cefc1cb31b50d4490d12"
2024-06-05T08:33:32.7706000Z 		},
2024-06-05T08:33:32.7706270Z 		{
2024-06-05T08:33:32.7706735Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7707367Z 			"size": 35867203,
2024-06-05T08:33:32.7707948Z 			"digest": "sha256:2d101f59465750f3b8295421045ff28c56f819a9d6aaff34dde4194f0f45884b"
2024-06-05T08:33:32.7708587Z 		},
2024-06-05T08:33:32.7708862Z 		{
2024-06-05T08:33:32.7709337Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7709925Z 			"size": 104,
2024-06-05T08:33:32.7710483Z 			"digest": "sha256:17c2c6432479b11d80b95131c4a9b289f433eef8c42c866ede757878721a4936"
2024-06-05T08:33:32.7711118Z 		},
2024-06-05T08:33:32.7711387Z 		{
2024-06-05T08:33:32.7711855Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7712439Z 			"size": 425,
2024-06-05T08:33:32.7712999Z 			"digest": "sha256:6b06891c48da4305bd3ee64d70c4445bf5b6525da0037e28e8d08835d6ea2b73"
2024-06-05T08:33:32.7713658Z 		},
2024-06-05T08:33:32.7713936Z 		{
2024-06-05T08:33:32.7714391Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7714982Z 			"size": 20262071,
2024-06-05T08:33:32.7715575Z 			"digest": "sha256:6d0905301e9b97d2ccbda5c1db919052eab495ce23978e5a9eda3111e3f90137"
2024-06-05T08:33:32.7716230Z 		},
2024-06-05T08:33:32.7716497Z 		{
2024-06-05T08:33:32.7716964Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7717550Z 			"size": 438,
2024-06-05T08:33:32.7718126Z 			"digest": "sha256:f65a0035d0f4a236d062a77fe5bb16f744622e84beab5ec0b31e71fa6b7cd520"
2024-06-05T08:33:32.7718772Z 		},
2024-06-05T08:33:32.7719040Z 		{
2024-06-05T08:33:32.7719504Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7720092Z 			"size": 701,
2024-06-05T08:33:32.7720657Z 			"digest": "sha256:0b14b049729cab9bdb836a987d36f5166f3308c620eb4bb48c7f6a98170bb6ef"
2024-06-05T08:33:32.7721311Z 		},
2024-06-05T08:33:32.7721591Z 		{
2024-06-05T08:33:32.7722155Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7722750Z 			"size": 141,
2024-06-05T08:33:32.7723342Z 			"digest": "sha256:f3ca65c25eed148e825ed0588df97d94cc9070d8b8f9b2eba0dc659bc1b451db"
2024-06-05T08:33:32.7724021Z 		},
2024-06-05T08:33:32.7724285Z 		{
2024-06-05T08:33:32.7724754Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7725428Z 			"size": 137,
2024-06-05T08:33:32.7726015Z 			"digest": "sha256:eb7f9159422dd91c3f2cc8057311475f3aa055dffc54a8bc7ebcdb8bf20052b4"
2024-06-05T08:33:32.7726845Z 		},
2024-06-05T08:33:32.7727120Z 		{
2024-06-05T08:33:32.7727716Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7728339Z 			"size": 4981032130,
2024-06-05T08:33:32.7728951Z 			"digest": "sha256:e29f705987b8280407813b5879bfdfa6019a34935be9600b8834301cd4998970"
2024-06-05T08:33:32.7729625Z 		},
2024-06-05T08:33:32.7729895Z 		{
2024-06-05T08:33:32.7730364Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7731057Z 			"size": 192,
2024-06-05T08:33:32.7731620Z 			"digest": "sha256:eaf1734e8825229c7695c88954c068ccc3b36c98175a2b6ad2e28379c587bc71"
2024-06-05T08:33:32.7732272Z 		},
2024-06-05T08:33:32.7732544Z 		{
2024-06-05T08:33:32.7733008Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7733592Z 			"size": 563,
2024-06-05T08:33:32.7734166Z 			"digest": "sha256:6e047fa4162056ff836b3af4d2c59f505a914b0bc7c012fd2a0d5e801e283042"
2024-06-05T08:33:32.7734825Z 		},
2024-06-05T08:33:32.7735094Z 		{
2024-06-05T08:33:32.7735553Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7736142Z 			"size": 43162767,
2024-06-05T08:33:32.7736744Z 			"digest": "sha256:8fbad97b0a92c6fceeb2a2a075d3de1aeb74d1610ae72856410066369de848db"
2024-06-05T08:33:32.7737401Z 		},
2024-06-05T08:33:32.7737669Z 		{
2024-06-05T08:33:32.7738133Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7738725Z 			"size": 106,
2024-06-05T08:33:32.7739296Z 			"digest": "sha256:0f804bdfa926acf90d109082d0dcabb8713a310c50b6f942250babdb7611ede0"
2024-06-05T08:33:32.7739948Z 		},
2024-06-05T08:33:32.7740217Z 		{
2024-06-05T08:33:32.7740678Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7741283Z 			"size": 1211,
2024-06-05T08:33:32.7741861Z 			"digest": "sha256:7a6b033ce69d9dacb0366b1e26e6831782b18a1e4e3e3207865e8a5544e5ef01"
2024-06-05T08:33:32.7742507Z 		},
2024-06-05T08:33:32.7742779Z 		{
2024-06-05T08:33:32.7743238Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7743823Z 			"size": 701,
2024-06-05T08:33:32.7744391Z 			"digest": "sha256:0b14b049729cab9bdb836a987d36f5166f3308c620eb4bb48c7f6a98170bb6ef"
2024-06-05T08:33:32.7745036Z 		},
2024-06-05T08:33:32.7745308Z 		{
2024-06-05T08:33:32.7745783Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7746374Z 			"size": 138,
2024-06-05T08:33:32.7746956Z 			"digest": "sha256:f8c51cc7f9eadd3dc962e62bd8b1ae024635df0bd063031ac1bae7f5bd9ff20c"
2024-06-05T08:33:32.7747624Z 		},
2024-06-05T08:33:32.7747895Z 		{
2024-06-05T08:33:32.7748358Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7748940Z 			"size": 118,
2024-06-05T08:33:32.7749509Z 			"digest": "sha256:7ec254469299b51f5917d32bd56e1fb6a8e6dd22216109bd592ef721b5db04f0"
2024-06-05T08:33:32.7750157Z 		},
2024-06-05T08:33:32.7750422Z 		{
2024-06-05T08:33:32.7750885Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7751477Z 			"size": 1855315830,
2024-06-05T08:33:32.7752092Z 			"digest": "sha256:cc17eb4ad79f94eb8e495d482eab1bd7636dbf6a12f42432ac08b2d8d66599f9"
2024-06-05T08:33:32.7752749Z 		},
2024-06-05T08:33:32.7753022Z 		{
2024-06-05T08:33:32.7753489Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7754074Z 			"size": 172,
2024-06-05T08:33:32.7754652Z 			"digest": "sha256:5f2ce86c63a4bbd4c79e9851a7d49d71afa8ff01c0d63be8178b760be640650a"
2024-06-05T08:33:32.7755315Z 		},
2024-06-05T08:33:32.7755585Z 		{
2024-06-05T08:33:32.7756049Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7756634Z 			"size": 1842,
2024-06-05T08:33:32.7757203Z 			"digest": "sha256:90fe1b39949401e51e3d00ac733602a35b0614de5a500a330d280bcd07fb0b29"
2024-06-05T08:33:32.7758020Z 		},
2024-06-05T08:33:32.7758290Z 		{
2024-06-05T08:33:32.7758764Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7759360Z 			"size": 7529792,
2024-06-05T08:33:32.7759954Z 			"digest": "sha256:967e565fba22173b2ebcdc832848b82b0a18a3e8a3fcbf6e7074090f3b48f75c"
2024-06-05T08:33:32.7760612Z 		},
2024-06-05T08:33:32.7760890Z 		{
2024-06-05T08:33:32.7761357Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7762076Z 			"size": 163,
2024-06-05T08:33:32.7762675Z 			"digest": "sha256:fad6ad34e48e2e0fb7bc863ebe6fbfa6ec6f6ad7d3e9d4b22e4744f0ffb0b811"
2024-06-05T08:33:32.7763406Z 		},
2024-06-05T08:33:32.7763677Z 		{
2024-06-05T08:33:32.7764134Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7764724Z 			"size": 7939,
2024-06-05T08:33:32.7765293Z 			"digest": "sha256:713e9e98f85c610568ac29ad851b43d25d7436ec15eea123fc17966939a6b78c"
2024-06-05T08:33:32.7765937Z 		},
2024-06-05T08:33:32.7766205Z 		{
2024-06-05T08:33:32.7766846Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7767450Z 			"size": 8072,
2024-06-05T08:33:32.7768038Z 			"digest": "sha256:ce4436966229bcee6cbc62bf33e7b35d4e06972d46e4d664dd245aa70daa84b2"
2024-06-05T08:33:32.7768700Z 		},
2024-06-05T08:33:32.7768972Z 		{
2024-06-05T08:33:32.7769443Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7770037Z 			"size": 304,
2024-06-05T08:33:32.7770607Z 			"digest": "sha256:bb2e4aa1f249503545ff621d364a77c047079b16318aafcf654df015efbdd8e7"
2024-06-05T08:33:32.7771276Z 		},
2024-06-05T08:33:32.7771547Z 		{
2024-06-05T08:33:32.7772019Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7772617Z 			"size": 7628733,
2024-06-05T08:33:32.7773214Z 			"digest": "sha256:0af89eb704c12eea8a4f889d957359d2dc4835e1828235c4bebc7a12c9daeb39"
2024-06-05T08:33:32.7773881Z 		},
2024-06-05T08:33:32.7774150Z 		{
2024-06-05T08:33:32.7774623Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7775218Z 			"size": 108,
2024-06-05T08:33:32.7775783Z 			"digest": "sha256:5fb77b92600205b8a76749635c62f254f8767fba907ba5777a19cf7b0b815774"
2024-06-05T08:33:32.7776428Z 		},
2024-06-05T08:33:32.7776703Z 		{
2024-06-05T08:33:32.7777172Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7777766Z 			"size": 54145482,
2024-06-05T08:33:32.7778372Z 			"digest": "sha256:ff46480ff616c5e907bfcb3e8e62d933f58bd61bdce9ee463038435c62f920dd"
2024-06-05T08:33:32.7779043Z 		},
2024-06-05T08:33:32.7779320Z 		{
2024-06-05T08:33:32.7779784Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7780375Z 			"size": 472,
2024-06-05T08:33:32.7780956Z 			"digest": "sha256:7cedf7d3de5c2e1604bee473147a487322d80d618e60356eafc860169456efc2"
2024-06-05T08:33:32.7781619Z 		},
2024-06-05T08:33:32.7781885Z 		{
2024-06-05T08:33:32.7782356Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7782949Z 			"size": 1374858584,
2024-06-05T08:33:32.7783561Z 			"digest": "sha256:fc169489ff75082c3bf4adf0668db87e3b3bddaed687f45f57a60565ef8618df"
2024-06-05T08:33:32.7784223Z 		},
2024-06-05T08:33:32.7784497Z 		{
2024-06-05T08:33:32.7784977Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7785564Z 			"size": 106,
2024-06-05T08:33:32.7786146Z 			"digest": "sha256:49f1c2d9f052ec654e5ec30f99a6d95fd89509a36914f34cadef6a5165c177ec"
2024-06-05T08:33:32.7786810Z 		},
2024-06-05T08:33:32.7787086Z 		{
2024-06-05T08:33:32.7787546Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7788147Z 			"size": 561,
2024-06-05T08:33:32.7788724Z 			"digest": "sha256:387dbdfd92815821601f69fc909a25daedf5f6865602fbaf00963a5f6e3f0d39"
2024-06-05T08:33:32.7789386Z 		},
2024-06-05T08:33:32.7789655Z 		{
2024-06-05T08:33:32.7790122Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7790839Z 			"size": 46248045,
2024-06-05T08:33:32.7791448Z 			"digest": "sha256:737652e4ef2cbfc902c7768f4ceb4b5e6c5cefb2a1a995594563ed17f3f747ea"
2024-06-05T08:33:32.7792113Z 		},
2024-06-05T08:33:32.7792385Z 		{
2024-06-05T08:33:32.7792851Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-06-05T08:33:32.7793440Z 			"size": 111,
2024-06-05T08:33:32.7794017Z 			"digest": "sha256:542fb1838ddb24f4af7aa9e3b46299ecbe12020d5dc2e883fee61261829b758f"
2024-06-05T08:33:32.7794680Z 		}
2024-06-05T08:33:32.7794952Z 	]
2024-06-05T08:33:32.7795213Z }
2024-06-05T08:33:32.7910857Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-06-05T08:33:32.7911368Z tag=${ECR_DOCKER_IMAGE##*/}
2024-06-05T08:33:32.7911956Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-06-05T08:33:32.7919765Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.7920299Z env:
2024-06-05T08:33:32.7920610Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.7921830Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.7923169Z ##[endgroup]
2024-06-05T08:33:32.7945388Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.7995841Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2024-06-05T08:33:32.7996455Z with:
2024-06-05T08:33:32.7997653Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.7999039Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.7999579Z env:
2024-06-05T08:33:32.7999889Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.8000267Z ##[endgroup]
2024-06-05T08:33:32.8016990Z ##[group]Run set -x
2024-06-05T08:33:32.8017356Z set -x
2024-06-05T08:33:32.8017688Z set +e
2024-06-05T08:33:32.8018013Z 
2024-06-05T08:33:32.8018316Z login() {
2024-06-05T08:33:32.8019021Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-06-05T08:33:32.8019778Z }
2024-06-05T08:33:32.8020082Z 
2024-06-05T08:33:32.8020411Z retry () {
2024-06-05T08:33:32.8020835Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-06-05T08:33:32.8021308Z }
2024-06-05T08:33:32.8021616Z 
2024-06-05T08:33:32.8021962Z retry login "${DOCKER_REGISTRY}"
2024-06-05T08:33:32.8022405Z 
2024-06-05T08:33:32.8022703Z set -e
2024-06-05T08:33:32.8023213Z # ignore output since only exit code is used for conditional
2024-06-05T08:33:32.8023957Z # only pull docker image if it's not available locally
2024-06-05T08:33:32.8024765Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-06-05T08:33:32.8025504Z   retry docker pull "${DOCKER_IMAGE}"
2024-06-05T08:33:32.8025986Z fi
2024-06-05T08:33:32.8032855Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:33:32.8033382Z env:
2024-06-05T08:33:32.8033697Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:33:32.8034897Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:32.8036255Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.8036808Z ##[endgroup]
2024-06-05T08:33:32.8057565Z + set +e
2024-06-05T08:33:32.8058215Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.8058885Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:32.8060032Z + aws ecr get-login-password --region us-east-1
2024-06-05T08:33:32.8060857Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-06-05T08:33:33.2262393Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-06-05T08:33:33.2263633Z Configure a credential helper to remove this warning. See
2024-06-05T08:33:33.2264878Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-06-05T08:33:33.2265634Z 
2024-06-05T08:33:33.2265812Z Login Succeeded
2024-06-05T08:33:33.2277981Z + set -e
2024-06-05T08:33:33.2279365Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:33.2434086Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:33.2436329Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:33:33.4686717Z 28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks
2024-06-05T08:33:33.4688255Z 63e9bbe32327: Pulling fs layer
2024-06-05T08:33:33.4688877Z cfb3d849840e: Pulling fs layer
2024-06-05T08:33:33.4689754Z 968831e596a6: Pulling fs layer
2024-06-05T08:33:33.4690357Z ea310eb267ca: Pulling fs layer
2024-06-05T08:33:33.4690931Z 3af11d09e9cd: Pulling fs layer
2024-06-05T08:33:33.4691379Z ebfec18059b9: Pulling fs layer
2024-06-05T08:33:33.4691815Z 533b4aebf169: Pulling fs layer
2024-06-05T08:33:33.4692234Z 9dd75d06a091: Pulling fs layer
2024-06-05T08:33:33.4692734Z 30bfca4dd349: Pulling fs layer
2024-06-05T08:33:33.4693211Z 1b57ce94cad9: Pulling fs layer
2024-06-05T08:33:33.4693618Z 9ee6bdb31195: Pulling fs layer
2024-06-05T08:33:33.4694104Z 70afc194f361: Pulling fs layer
2024-06-05T08:33:33.4694561Z a15b9861c14e: Pulling fs layer
2024-06-05T08:33:33.4695034Z bcc9c2b688f4: Pulling fs layer
2024-06-05T08:33:33.4695464Z 9bdedd1ac09a: Pulling fs layer
2024-06-05T08:33:33.4695922Z ebfec18059b9: Waiting
2024-06-05T08:33:33.4696302Z 0d1cc19d82a4: Pulling fs layer
2024-06-05T08:33:33.4696762Z d5495d103885: Pulling fs layer
2024-06-05T08:33:33.4697179Z ea09054a5a20: Pulling fs layer
2024-06-05T08:33:33.4697653Z fbcdf69e3d41: Pulling fs layer
2024-06-05T08:33:33.4698047Z 533b4aebf169: Waiting
2024-06-05T08:33:33.4698403Z 0b14b049729c: Pulling fs layer
2024-06-05T08:33:33.4698790Z 9dd75d06a091: Waiting
2024-06-05T08:33:33.4699158Z 465d7dda2107: Pulling fs layer
2024-06-05T08:33:33.4699557Z 950fe48ec990: Pulling fs layer
2024-06-05T08:33:33.4699960Z 4d2f5b21f13f: Pulling fs layer
2024-06-05T08:33:33.4700351Z bcc9c2b688f4: Waiting
2024-06-05T08:33:33.4700716Z 5e6df7aeac96: Pulling fs layer
2024-06-05T08:33:33.4701102Z a15b9861c14e: Waiting
2024-06-05T08:33:33.4701452Z 30bfca4dd349: Waiting
2024-06-05T08:33:33.4701806Z 9bdedd1ac09a: Waiting
2024-06-05T08:33:33.4702162Z 76150770c77c: Pulling fs layer
2024-06-05T08:33:33.4702571Z d076da148a13: Pulling fs layer
2024-06-05T08:33:33.4702958Z 70afc194f361: Waiting
2024-06-05T08:33:33.4703315Z 8517857fd1ee: Pulling fs layer
2024-06-05T08:33:33.4703700Z 9ee6bdb31195: Waiting
2024-06-05T08:33:33.4704049Z 1b57ce94cad9: Waiting
2024-06-05T08:33:33.4704460Z 7e68d7c69ce8: Pulling fs layer
2024-06-05T08:33:33.4704886Z c3ad3f0969a2: Pulling fs layer
2024-06-05T08:33:33.4705286Z 4b05a1fe871d: Pulling fs layer
2024-06-05T08:33:33.4705693Z 7f2ff6e7943b: Pulling fs layer
2024-06-05T08:33:33.4706145Z fcf3589d3f0d: Pulling fs layer
2024-06-05T08:33:33.4706562Z d5495d103885: Waiting
2024-06-05T08:33:33.4706916Z 3af11d09e9cd: Waiting
2024-06-05T08:33:33.4707273Z 950fe48ec990: Waiting
2024-06-05T08:33:33.4707620Z ea09054a5a20: Waiting
2024-06-05T08:33:33.4707962Z 4d2f5b21f13f: Waiting
2024-06-05T08:33:33.4708304Z 8517857fd1ee: Waiting
2024-06-05T08:33:33.4708686Z 7e68d7c69ce8: Waiting
2024-06-05T08:33:33.4709035Z 5e6df7aeac96: Waiting
2024-06-05T08:33:33.4709381Z fbcdf69e3d41: Waiting
2024-06-05T08:33:33.4709727Z d076da148a13: Waiting
2024-06-05T08:33:33.4710066Z 0b14b049729c: Waiting
2024-06-05T08:33:33.4710398Z 465d7dda2107: Waiting
2024-06-05T08:33:33.4710745Z ea310eb267ca: Waiting
2024-06-05T08:33:33.4711092Z 7f2ff6e7943b: Waiting
2024-06-05T08:33:33.4711438Z c3ad3f0969a2: Waiting
2024-06-05T08:33:33.4711793Z 4b05a1fe871d: Waiting
2024-06-05T08:33:33.4712133Z 76150770c77c: Waiting
2024-06-05T08:33:33.4712474Z fcf3589d3f0d: Waiting
2024-06-05T08:33:33.4712830Z bb7540904742: Pulling fs layer
2024-06-05T08:33:33.4713380Z b1419f097390: Pulling fs layer
2024-06-05T08:33:33.4713790Z fe30e7b83e01: Pulling fs layer
2024-06-05T08:33:33.4714193Z 2d101f594657: Pulling fs layer
2024-06-05T08:33:33.4714586Z 17c2c6432479: Pulling fs layer
2024-06-05T08:33:33.4714987Z 6b06891c48da: Pulling fs layer
2024-06-05T08:33:33.4715392Z 6d0905301e9b: Pulling fs layer
2024-06-05T08:33:33.4715792Z f65a0035d0f4: Pulling fs layer
2024-06-05T08:33:33.4716169Z b1419f097390: Waiting
2024-06-05T08:33:33.4716513Z fe30e7b83e01: Waiting
2024-06-05T08:33:33.4716875Z f3ca65c25eed: Pulling fs layer
2024-06-05T08:33:33.4717257Z 2d101f594657: Waiting
2024-06-05T08:33:33.4717600Z 6d0905301e9b: Waiting
2024-06-05T08:33:33.4717954Z 6b06891c48da: Waiting
2024-06-05T08:33:33.4718332Z f65a0035d0f4: Waiting
2024-06-05T08:33:33.4718875Z eb7f9159422d: Pulling fs layer
2024-06-05T08:33:33.4719336Z e29f705987b8: Pulling fs layer
2024-06-05T08:33:33.4719746Z eaf1734e8825: Pulling fs layer
2024-06-05T08:33:33.4720141Z 6e047fa41620: Pulling fs layer
2024-06-05T08:33:33.4720555Z 8fbad97b0a92: Pulling fs layer
2024-06-05T08:33:33.4720999Z 0f804bdfa926: Pulling fs layer
2024-06-05T08:33:33.4721439Z 7a6b033ce69d: Pulling fs layer
2024-06-05T08:33:33.4721839Z f8c51cc7f9ea: Pulling fs layer
2024-06-05T08:33:33.4722394Z 7ec254469299: Pulling fs layer
2024-06-05T08:33:33.4722785Z f3ca65c25eed: Waiting
2024-06-05T08:33:33.4723136Z f8c51cc7f9ea: Waiting
2024-06-05T08:33:33.4723476Z eb7f9159422d: Waiting
2024-06-05T08:33:33.4723852Z cc17eb4ad79f: Pulling fs layer
2024-06-05T08:33:33.4724243Z 7a6b033ce69d: Waiting
2024-06-05T08:33:33.4724586Z e29f705987b8: Waiting
2024-06-05T08:33:33.4724938Z 5f2ce86c63a4: Pulling fs layer
2024-06-05T08:33:33.4725327Z eaf1734e8825: Waiting
2024-06-05T08:33:33.4725670Z 0f804bdfa926: Waiting
2024-06-05T08:33:33.4726039Z 90fe1b399494: Pulling fs layer
2024-06-05T08:33:33.4726553Z 7ec254469299: Waiting
2024-06-05T08:33:33.4726911Z 6e047fa41620: Waiting
2024-06-05T08:33:33.4727294Z 967e565fba22: Pulling fs layer
2024-06-05T08:33:33.4727708Z cc17eb4ad79f: Waiting
2024-06-05T08:33:33.4728072Z fad6ad34e48e: Pulling fs layer
2024-06-05T08:33:33.4728478Z 713e9e98f85c: Pulling fs layer
2024-06-05T08:33:33.4728877Z ce4436966229: Pulling fs layer
2024-06-05T08:33:33.4729276Z bb2e4aa1f249: Pulling fs layer
2024-06-05T08:33:33.4729666Z 8fbad97b0a92: Waiting
2024-06-05T08:33:33.4730013Z fad6ad34e48e: Waiting
2024-06-05T08:33:33.4730371Z 0af89eb704c1: Pulling fs layer
2024-06-05T08:33:33.4730780Z 5fb77b926002: Pulling fs layer
2024-06-05T08:33:33.4731183Z ff46480ff616: Pulling fs layer
2024-06-05T08:33:33.4731589Z 7cedf7d3de5c: Pulling fs layer
2024-06-05T08:33:33.4731974Z 0af89eb704c1: Waiting
2024-06-05T08:33:33.4732342Z fc169489ff75: Pulling fs layer
2024-06-05T08:33:33.4732732Z 5fb77b926002: Waiting
2024-06-05T08:33:33.4733098Z 49f1c2d9f052: Pulling fs layer
2024-06-05T08:33:33.4733486Z bb2e4aa1f249: Waiting
2024-06-05T08:33:33.4733851Z 387dbdfd9281: Pulling fs layer
2024-06-05T08:33:33.4734304Z 737652e4ef2c: Pulling fs layer
2024-06-05T08:33:33.4734700Z 7cedf7d3de5c: Waiting
2024-06-05T08:33:33.4735052Z 5f2ce86c63a4: Waiting
2024-06-05T08:33:33.4735419Z 542fb1838ddb: Pulling fs layer
2024-06-05T08:33:33.4735812Z fc169489ff75: Waiting
2024-06-05T08:33:33.4736150Z 49f1c2d9f052: Waiting
2024-06-05T08:33:33.4736497Z 387dbdfd9281: Waiting
2024-06-05T08:33:33.6131036Z cfb3d849840e: Verifying Checksum
2024-06-05T08:33:33.6131697Z cfb3d849840e: Download complete
2024-06-05T08:33:33.6947247Z ea310eb267ca: Verifying Checksum
2024-06-05T08:33:33.6947716Z ea310eb267ca: Download complete
2024-06-05T08:33:33.7743657Z 3af11d09e9cd: Verifying Checksum
2024-06-05T08:33:33.7744171Z 3af11d09e9cd: Download complete
2024-06-05T08:33:33.8111102Z 63e9bbe32327: Verifying Checksum
2024-06-05T08:33:33.8111558Z 63e9bbe32327: Download complete
2024-06-05T08:33:33.9003537Z 533b4aebf169: Download complete
2024-06-05T08:33:33.9802534Z 9dd75d06a091: Verifying Checksum
2024-06-05T08:33:33.9803073Z 9dd75d06a091: Download complete
2024-06-05T08:33:34.0562119Z 30bfca4dd349: Verifying Checksum
2024-06-05T08:33:34.0563070Z 30bfca4dd349: Download complete
2024-06-05T08:33:34.1042600Z 968831e596a6: Verifying Checksum
2024-06-05T08:33:34.1043143Z 968831e596a6: Download complete
2024-06-05T08:33:34.1854851Z 9ee6bdb31195: Verifying Checksum
2024-06-05T08:33:34.1855959Z 9ee6bdb31195: Download complete
2024-06-05T08:33:34.2866803Z 70afc194f361: Verifying Checksum
2024-06-05T08:33:34.2867540Z 70afc194f361: Download complete
2024-06-05T08:33:34.4739220Z 63e9bbe32327: Pull complete
2024-06-05T08:33:34.7051817Z cfb3d849840e: Pull complete
2024-06-05T08:33:35.6270746Z 968831e596a6: Pull complete
2024-06-05T08:33:35.7203131Z ea310eb267ca: Pull complete
2024-06-05T08:33:35.8241293Z 3af11d09e9cd: Pull complete
2024-06-05T08:33:36.8074611Z a15b9861c14e: Verifying Checksum
2024-06-05T08:33:36.8075405Z a15b9861c14e: Download complete
2024-06-05T08:33:36.8805122Z bcc9c2b688f4: Verifying Checksum
2024-06-05T08:33:36.8805647Z bcc9c2b688f4: Download complete
2024-06-05T08:33:36.9697135Z 9bdedd1ac09a: Verifying Checksum
2024-06-05T08:33:36.9697612Z 9bdedd1ac09a: Download complete
2024-06-05T08:33:37.0500568Z 0d1cc19d82a4: Verifying Checksum
2024-06-05T08:33:37.0501091Z 0d1cc19d82a4: Download complete
2024-06-05T08:33:38.0323799Z d5495d103885: Verifying Checksum
2024-06-05T08:33:38.0324325Z d5495d103885: Download complete
2024-06-05T08:33:38.1398521Z ea09054a5a20: Verifying Checksum
2024-06-05T08:33:38.1398977Z ea09054a5a20: Download complete
2024-06-05T08:33:38.2147897Z fbcdf69e3d41: Verifying Checksum
2024-06-05T08:33:38.2148722Z fbcdf69e3d41: Download complete
2024-06-05T08:33:38.3007826Z 0b14b049729c: Verifying Checksum
2024-06-05T08:33:38.3008272Z 0b14b049729c: Download complete
2024-06-05T08:33:47.4491989Z ebfec18059b9: Verifying Checksum
2024-06-05T08:33:47.4492502Z ebfec18059b9: Download complete
2024-06-05T08:33:47.5303108Z 950fe48ec990: Verifying Checksum
2024-06-05T08:33:47.6347858Z 950fe48ec990: Download complete
2024-06-05T08:33:47.6348338Z 4d2f5b21f13f: Download complete
2024-06-05T08:33:47.7122917Z 5e6df7aeac96: Verifying Checksum
2024-06-05T08:33:47.7123457Z 5e6df7aeac96: Download complete
2024-06-05T08:33:47.7903964Z 76150770c77c: Verifying Checksum
2024-06-05T08:33:47.7904473Z 76150770c77c: Download complete
2024-06-05T08:33:47.8695654Z d076da148a13: Download complete
2024-06-05T08:33:49.1481015Z 8517857fd1ee: Verifying Checksum
2024-06-05T08:33:49.1481672Z 8517857fd1ee: Download complete
2024-06-05T08:33:49.2196446Z 7e68d7c69ce8: Download complete
2024-06-05T08:33:49.3008026Z c3ad3f0969a2: Verifying Checksum
2024-06-05T08:33:49.3008646Z c3ad3f0969a2: Download complete
2024-06-05T08:33:49.3974302Z 4b05a1fe871d: Download complete
2024-06-05T08:33:49.4688706Z 7f2ff6e7943b: Verifying Checksum
2024-06-05T08:33:49.4689477Z 7f2ff6e7943b: Download complete
2024-06-05T08:33:49.5488086Z fcf3589d3f0d: Verifying Checksum
2024-06-05T08:33:49.5488721Z fcf3589d3f0d: Download complete
2024-06-05T08:33:54.0682739Z bb7540904742: Verifying Checksum
2024-06-05T08:33:54.0683412Z bb7540904742: Download complete
2024-06-05T08:33:54.1518778Z b1419f097390: Verifying Checksum
2024-06-05T08:33:54.1519402Z b1419f097390: Download complete
2024-06-05T08:33:54.2357570Z fe30e7b83e01: Verifying Checksum
2024-06-05T08:33:54.2358050Z fe30e7b83e01: Download complete
2024-06-05T08:33:54.6565880Z 2d101f594657: Verifying Checksum
2024-06-05T08:33:54.6566670Z 2d101f594657: Download complete
2024-06-05T08:33:54.7450875Z 17c2c6432479: Verifying Checksum
2024-06-05T08:33:54.7451351Z 17c2c6432479: Download complete
2024-06-05T08:33:54.8336793Z 6b06891c48da: Download complete
2024-06-05T08:33:55.1022405Z 6d0905301e9b: Verifying Checksum
2024-06-05T08:33:55.1022915Z 6d0905301e9b: Download complete
2024-06-05T08:33:55.1806288Z f65a0035d0f4: Verifying Checksum
2024-06-05T08:33:55.1806953Z f65a0035d0f4: Download complete
2024-06-05T08:33:55.2669282Z f3ca65c25eed: Verifying Checksum
2024-06-05T08:33:55.2669805Z f3ca65c25eed: Download complete
2024-06-05T08:33:55.3481277Z eb7f9159422d: Verifying Checksum
2024-06-05T08:33:55.3481836Z eb7f9159422d: Download complete
2024-06-05T08:33:59.6227794Z ebfec18059b9: Pull complete
2024-06-05T08:33:59.8706960Z 533b4aebf169: Pull complete
2024-06-05T08:34:00.1052564Z 9dd75d06a091: Pull complete
2024-06-05T08:34:00.2007011Z 1b57ce94cad9: Verifying Checksum
2024-06-05T08:34:00.2007519Z 1b57ce94cad9: Download complete
2024-06-05T08:34:00.3325534Z eaf1734e8825: Verifying Checksum
2024-06-05T08:34:00.3326031Z eaf1734e8825: Download complete
2024-06-05T08:34:00.3633681Z 30bfca4dd349: Pull complete
2024-06-05T08:34:00.4052152Z 6e047fa41620: Verifying Checksum
2024-06-05T08:34:00.4052604Z 6e047fa41620: Download complete
2024-06-05T08:34:00.8858715Z 8fbad97b0a92: Verifying Checksum
2024-06-05T08:34:00.8859323Z 8fbad97b0a92: Download complete
2024-06-05T08:34:00.9674673Z 0f804bdfa926: Download complete
2024-06-05T08:34:01.0406843Z 7a6b033ce69d: Verifying Checksum
2024-06-05T08:34:01.0407451Z 7a6b033ce69d: Download complete
2024-06-05T08:34:01.1203023Z f8c51cc7f9ea: Verifying Checksum
2024-06-05T08:34:01.1203600Z f8c51cc7f9ea: Download complete
2024-06-05T08:34:01.2008824Z 7ec254469299: Verifying Checksum
2024-06-05T08:34:01.2009430Z 7ec254469299: Download complete
2024-06-05T08:34:06.3365794Z 465d7dda2107: Verifying Checksum
2024-06-05T08:34:06.3366299Z 465d7dda2107: Download complete
2024-06-05T08:34:06.4152130Z 5f2ce86c63a4: Verifying Checksum
2024-06-05T08:34:06.4152651Z 5f2ce86c63a4: Download complete
2024-06-05T08:34:06.4918646Z 90fe1b399494: Download complete
2024-06-05T08:34:06.6282087Z 967e565fba22: Verifying Checksum
2024-06-05T08:34:06.6282610Z 967e565fba22: Download complete
2024-06-05T08:34:06.7156941Z fad6ad34e48e: Verifying Checksum
2024-06-05T08:34:06.7157413Z fad6ad34e48e: Download complete
2024-06-05T08:34:06.7864861Z 713e9e98f85c: Verifying Checksum
2024-06-05T08:34:06.7865327Z 713e9e98f85c: Download complete
2024-06-05T08:34:06.8678568Z ce4436966229: Download complete
2024-06-05T08:34:06.9551580Z bb2e4aa1f249: Download complete
2024-06-05T08:34:07.1055044Z 0af89eb704c1: Verifying Checksum
2024-06-05T08:34:07.1055550Z 0af89eb704c1: Download complete
2024-06-05T08:34:07.1844714Z 5fb77b926002: Verifying Checksum
2024-06-05T08:34:07.1845235Z 5fb77b926002: Download complete
2024-06-05T08:34:07.7923532Z ff46480ff616: Verifying Checksum
2024-06-05T08:34:07.7924052Z ff46480ff616: Download complete
2024-06-05T08:34:07.8711584Z 7cedf7d3de5c: Verifying Checksum
2024-06-05T08:34:07.8712116Z 7cedf7d3de5c: Download complete
2024-06-05T08:34:34.0030321Z cc17eb4ad79f: Verifying Checksum
2024-06-05T08:34:34.0032571Z cc17eb4ad79f: Download complete
2024-06-05T08:34:34.0968751Z 49f1c2d9f052: Verifying Checksum
2024-06-05T08:34:34.0969383Z 49f1c2d9f052: Download complete
2024-06-05T08:34:34.1910907Z 387dbdfd9281: Download complete
2024-06-05T08:34:35.5372474Z 737652e4ef2c: Verifying Checksum
2024-06-05T08:34:35.5372993Z 737652e4ef2c: Download complete
2024-06-05T08:34:35.6191204Z 542fb1838ddb: Download complete
2024-06-05T08:34:39.3346419Z fc169489ff75: Verifying Checksum
2024-06-05T08:34:39.3346889Z fc169489ff75: Download complete
2024-06-05T08:35:01.7906869Z e29f705987b8: Verifying Checksum
2024-06-05T08:35:01.7907443Z e29f705987b8: Download complete
2024-06-05T08:35:18.7861465Z 1b57ce94cad9: Pull complete
2024-06-05T08:35:19.0225861Z 9ee6bdb31195: Pull complete
2024-06-05T08:35:19.2450488Z 70afc194f361: Pull complete
2024-06-05T08:35:24.4040456Z a15b9861c14e: Pull complete
2024-06-05T08:35:24.6284636Z bcc9c2b688f4: Pull complete
2024-06-05T08:35:24.8470503Z 9bdedd1ac09a: Pull complete
2024-06-05T08:35:25.0964282Z 0d1cc19d82a4: Pull complete
2024-06-05T08:35:26.9646299Z d5495d103885: Pull complete
2024-06-05T08:35:27.1560200Z ea09054a5a20: Pull complete
2024-06-05T08:35:27.3931612Z fbcdf69e3d41: Pull complete
2024-06-05T08:35:27.6439184Z 0b14b049729c: Pull complete
2024-06-05T08:36:04.6511476Z 465d7dda2107: Pull complete
2024-06-05T08:36:04.8378094Z 950fe48ec990: Pull complete
2024-06-05T08:36:04.9930780Z 4d2f5b21f13f: Pull complete
2024-06-05T08:36:05.2273967Z 5e6df7aeac96: Pull complete
2024-06-05T08:36:05.3700067Z 76150770c77c: Pull complete
2024-06-05T08:36:05.5312300Z d076da148a13: Pull complete
2024-06-05T08:36:08.6264449Z 8517857fd1ee: Pull complete
2024-06-05T08:36:08.8226572Z 7e68d7c69ce8: Pull complete
2024-06-05T08:36:09.0527075Z c3ad3f0969a2: Pull complete
2024-06-05T08:36:09.2656330Z 4b05a1fe871d: Pull complete
2024-06-05T08:36:09.4877449Z 7f2ff6e7943b: Pull complete
2024-06-05T08:36:09.7100670Z fcf3589d3f0d: Pull complete
2024-06-05T08:36:15.5679346Z bb7540904742: Pull complete
2024-06-05T08:36:15.7877986Z b1419f097390: Pull complete
2024-06-05T08:36:16.0242558Z fe30e7b83e01: Pull complete
2024-06-05T08:36:16.9301137Z 2d101f594657: Pull complete
2024-06-05T08:36:17.1868967Z 17c2c6432479: Pull complete
2024-06-05T08:36:17.3868585Z 6b06891c48da: Pull complete
2024-06-05T08:36:17.8209331Z 6d0905301e9b: Pull complete
2024-06-05T08:36:18.0393620Z f65a0035d0f4: Pull complete
2024-06-05T08:36:18.4504633Z f3ca65c25eed: Pull complete
2024-06-05T08:36:18.5933853Z eb7f9159422d: Pull complete
2024-06-05T08:36:55.5605244Z e29f705987b8: Pull complete
2024-06-05T08:36:55.7970285Z eaf1734e8825: Pull complete
2024-06-05T08:36:55.9954167Z 6e047fa41620: Pull complete
2024-06-05T08:36:57.3178713Z 8fbad97b0a92: Pull complete
2024-06-05T08:36:57.5638876Z 0f804bdfa926: Pull complete
2024-06-05T08:36:57.7680474Z 7a6b033ce69d: Pull complete
2024-06-05T08:36:58.1206607Z f8c51cc7f9ea: Pull complete
2024-06-05T08:36:58.3650580Z 7ec254469299: Pull complete
2024-06-05T08:37:28.0802768Z cc17eb4ad79f: Pull complete
2024-06-05T08:37:28.3048017Z 5f2ce86c63a4: Pull complete
2024-06-05T08:37:28.5229385Z 90fe1b399494: Pull complete
2024-06-05T08:37:28.8792166Z 967e565fba22: Pull complete
2024-06-05T08:37:29.0997692Z fad6ad34e48e: Pull complete
2024-06-05T08:37:29.2983838Z 713e9e98f85c: Pull complete
2024-06-05T08:37:29.5261886Z ce4436966229: Pull complete
2024-06-05T08:37:29.7459765Z bb2e4aa1f249: Pull complete
2024-06-05T08:37:30.3959673Z 0af89eb704c1: Pull complete
2024-06-05T08:37:30.6166808Z 5fb77b926002: Pull complete
2024-06-05T08:37:32.3879920Z ff46480ff616: Pull complete
2024-06-05T08:37:32.6259448Z 7cedf7d3de5c: Pull complete
2024-06-05T08:37:45.8611053Z fc169489ff75: Pull complete
2024-06-05T08:37:46.0633901Z 49f1c2d9f052: Pull complete
2024-06-05T08:37:46.2922049Z 387dbdfd9281: Pull complete
2024-06-05T08:37:47.1145670Z 737652e4ef2c: Pull complete
2024-06-05T08:37:47.3161215Z 542fb1838ddb: Pull complete
2024-06-05T08:37:47.4341857Z Digest: sha256:30399d85d4241c3b46fc1a06bbd1a5a4a40523c847a85377b045014c9718879c
2024-06-05T08:37:47.4662616Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:37:47.4807050Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:37:47.4865188Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-06-05T08:37:47.4866177Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-06-05T08:37:47.4874172Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:37:47.4874720Z env:
2024-06-05T08:37:47.4875029Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:37:47.4875411Z ##[endgroup]
2024-06-05T08:37:47.5034724Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2024-06-05T08:37:47.5035310Z with:
2024-06-05T08:37:47.5035627Z   driver-version: 550.54.15
2024-06-05T08:37:47.5035998Z env:
2024-06-05T08:37:47.5036296Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:37:47.5036671Z ##[endgroup]
2024-06-05T08:37:47.5134005Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-06-05T08:37:47.5134619Z with:
2024-06-05T08:37:47.5134945Z   timeout_minutes: 10
2024-06-05T08:37:47.5135323Z   max_attempts: 3
2024-06-05T08:37:47.5167513Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

2024-06-05T08:37:47.5200175Z   retry_wait_seconds: 10
2024-06-05T08:37:47.5200595Z   polling_interval_seconds: 1
2024-06-05T08:37:47.5201016Z   warning_on_retry: true
2024-06-05T08:37:47.5201425Z   continue_on_error: false
2024-06-05T08:37:47.5201821Z env:
2024-06-05T08:37:47.5202233Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:37:47.5202634Z   DRIVER_VERSION: 550.54.15
2024-06-05T08:37:47.5203029Z ##[endgroup]
2024-06-05T08:37:47.5727509Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-06-05T08:37:47.5728344Z + pre_install_nvidia_driver_amzn2
2024-06-05T08:37:47.5728907Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-06-05T08:37:47.8833997Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-06-05T08:37:47.9229624Z No Match for argument: nvidia-driver-latest-dkms
2024-06-05T08:37:47.9467277Z No Packages marked for removal
2024-06-05T08:37:47.9590183Z + install_nvidia_driver_common
2024-06-05T08:37:47.9593107Z + echo 'Before installing NVIDIA driver'
2024-06-05T08:37:47.9593599Z + lspci
2024-06-05T08:37:47.9594821Z Before installing NVIDIA driver
2024-06-05T08:37:47.9677602Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-06-05T08:37:47.9678643Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-06-05T08:37:47.9680098Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-06-05T08:37:47.9681309Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-06-05T08:37:47.9682799Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061
2024-06-05T08:37:47.9684008Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-06-05T08:37:47.9685139Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1)
2024-06-05T08:37:47.9686361Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-06-05T08:37:47.9687542Z + lsmod
2024-06-05T08:37:47.9695027Z Module                  Size  Used by
2024-06-05T08:37:47.9695691Z ib_core               266240  0
2024-06-05T08:37:47.9696265Z xt_nat                 16384  0
2024-06-05T08:37:47.9696853Z nvidia_modeset       1339392  0
2024-06-05T08:37:47.9697441Z veth                   16384  0
2024-06-05T08:37:47.9698030Z nvidia_uvm           4599808  0
2024-06-05T08:37:47.9698751Z nvidia              53989376  2 nvidia_uvm,nvidia_modeset
2024-06-05T08:37:47.9699527Z drm                   421888  1 nvidia
2024-06-05T08:37:47.9700195Z i2c_core               77824  2 nvidia,drm
2024-06-05T08:37:47.9701221Z backlight              16384  1 nvidia_modeset
2024-06-05T08:37:47.9701921Z xt_conntrack           16384  1
2024-06-05T08:37:47.9702525Z ipt_MASQUERADE         16384  1
2024-06-05T08:37:47.9703232Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2024-06-05T08:37:47.9703975Z nf_conntrack_netlink    49152  0
2024-06-05T08:37:47.9704657Z nfnetlink              16384  2 nf_conntrack_netlink
2024-06-05T08:37:47.9705356Z xfrm_user              45056  1
2024-06-05T08:37:47.9705977Z xfrm_algo              16384  1 xfrm_user
2024-06-05T08:37:47.9706650Z iptable_nat            16384  1
2024-06-05T08:37:47.9707270Z nf_conntrack_ipv4      16384  3
2024-06-05T08:37:47.9707940Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2024-06-05T08:37:47.9708990Z nf_nat_ipv4            16384  1 iptable_nat
2024-06-05T08:37:47.9709861Z nf_nat                 32768  3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat
2024-06-05T08:37:47.9711390Z nf_conntrack          155648  8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink
2024-06-05T08:37:47.9712651Z xt_addrtype            16384  2
2024-06-05T08:37:47.9713245Z iptable_filter         16384  1
2024-06-05T08:37:47.9713832Z br_netfilter           24576  0
2024-06-05T08:37:47.9714451Z bridge                172032  1 br_netfilter
2024-06-05T08:37:47.9715108Z stp                    16384  1 bridge
2024-06-05T08:37:47.9715726Z llc                    16384  2 bridge,stp
2024-06-05T08:37:47.9716350Z overlay                86016  0
2024-06-05T08:37:47.9716856Z sunrpc                393216  1
2024-06-05T08:37:47.9717257Z dm_mirror              28672  0
2024-06-05T08:37:47.9717732Z dm_region_hash         20480  1 dm_mirror
2024-06-05T08:37:47.9718273Z dm_log                 20480  2 dm_region_hash,dm_mirror
2024-06-05T08:37:47.9718925Z dm_mod                143360  2 dm_log,dm_mirror
2024-06-05T08:37:47.9719400Z dax                    69632  1 dm_mod
2024-06-05T08:37:47.9719836Z crc32_pclmul           16384  0
2024-06-05T08:37:47.9720267Z ghash_clmulni_intel    16384  0
2024-06-05T08:37:47.9720686Z pcbc                   16384  0
2024-06-05T08:37:47.9721084Z aesni_intel           188416  0
2024-06-05T08:37:47.9721512Z aes_x86_64             20480  1 aesni_intel
2024-06-05T08:37:47.9721994Z crypto_simd            16384  1 aesni_intel
2024-06-05T08:37:47.9722637Z glue_helper            16384  1 aesni_intel
2024-06-05T08:37:47.9723244Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2024-06-05T08:37:47.9723831Z mousedev               24576  0
2024-06-05T08:37:47.9724262Z evdev                  20480  3
2024-06-05T08:37:47.9724667Z psmouse                32768  0
2024-06-05T08:37:47.9725060Z button                 16384  0
2024-06-05T08:37:47.9725464Z ena                   139264  0
2024-06-05T08:37:47.9725866Z ptp                    20480  1 ena
2024-06-05T08:37:47.9726289Z pps_core               20480  1 ptp
2024-06-05T08:37:47.9726968Z crc32c_intel           24576  0
2024-06-05T08:37:47.9727426Z autofs4                49152  2
2024-06-05T08:37:47.9727979Z + modinfo nvidia
2024-06-05T08:37:47.9728762Z filename:       /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2024-06-05T08:37:47.9729523Z alias:          char-major-195-*
2024-06-05T08:37:47.9729946Z version:        550.54.15
2024-06-05T08:37:47.9730329Z supported:      external
2024-06-05T08:37:47.9730706Z license:        NVIDIA
2024-06-05T08:37:47.9731127Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-06-05T08:37:47.9731657Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-06-05T08:37:47.9732159Z srcversion:     833721318DA517F0C2FEC97
2024-06-05T08:37:47.9732664Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-06-05T08:37:47.9733217Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-06-05T08:37:47.9733755Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-06-05T08:37:47.9734281Z depends:        i2c-core,drm
2024-06-05T08:37:47.9734672Z retpoline:      Y
2024-06-05T08:37:47.9735012Z name:           nvidia
2024-06-05T08:37:47.9735751Z vermagic:       4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 
2024-06-05T08:37:47.9736471Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-06-05T08:37:47.9737180Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-06-05T08:37:47.9737834Z parm:           NVreg_ResmanDebugLevel:int
2024-06-05T08:37:47.9738328Z parm:           NVreg_RmLogonRC:int
2024-06-05T08:37:47.9738800Z parm:           NVreg_ModifyDeviceFiles:int
2024-06-05T08:37:47.9739303Z parm:           NVreg_DeviceFileUID:int
2024-06-05T08:37:47.9739787Z parm:           NVreg_DeviceFileGID:int
2024-06-05T08:37:47.9740269Z parm:           NVreg_DeviceFileMode:int
2024-06-05T08:37:47.9741176Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-06-05T08:37:47.9741802Z parm:           NVreg_UsePageAttributeTable:int
2024-06-05T08:37:47.9742328Z parm:           NVreg_EnablePCIeGen3:int
2024-06-05T08:37:47.9742802Z parm:           NVreg_EnableMSI:int
2024-06-05T08:37:47.9743264Z parm:           NVreg_TCEBypassMode:int
2024-06-05T08:37:47.9743766Z parm:           NVreg_EnableStreamMemOPs:int
2024-06-05T08:37:47.9744349Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-06-05T08:37:47.9745007Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-06-05T08:37:47.9745611Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-06-05T08:37:47.9746280Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-06-05T08:37:47.9746935Z parm:           NVreg_DynamicPowerManagement:int
2024-06-05T08:37:47.9747606Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-06-05T08:37:47.9748247Z parm:           NVreg_EnableGpuFirmware:int
2024-06-05T08:37:47.9748800Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-06-05T08:37:47.9749385Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-06-05T08:37:47.9749990Z parm:           NVreg_EnableUserNUMAManagement:int
2024-06-05T08:37:47.9750529Z parm:           NVreg_MemoryPoolSize:int
2024-06-05T08:37:47.9751048Z parm:           NVreg_KMallocHeapMaxSize:int
2024-06-05T08:37:47.9751580Z parm:           NVreg_VMallocHeapMaxSize:int
2024-06-05T08:37:47.9752105Z parm:           NVreg_IgnoreMMIOCheck:int
2024-06-05T08:37:47.9752592Z parm:           NVreg_NvLinkDisable:int
2024-06-05T08:37:47.9753146Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-06-05T08:37:47.9753723Z parm:           NVreg_RegisterPCIDriver:int
2024-06-05T08:37:47.9754250Z parm:           NVreg_EnableResizableBar:int
2024-06-05T08:37:47.9754777Z parm:           NVreg_EnableDbgBreakpoint:int
2024-06-05T08:37:47.9755325Z parm:           NVreg_EnableNonblockingOpen:int
2024-06-05T08:37:47.9755859Z parm:           NVreg_RegistryDwords:charp
2024-06-05T08:37:47.9756409Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-06-05T08:37:47.9756938Z parm:           NVreg_RmMsg:charp
2024-06-05T08:37:47.9757392Z parm:           NVreg_GpuBlacklist:charp
2024-06-05T08:37:47.9757916Z parm:           NVreg_TemporaryFilePath:charp
2024-06-05T08:37:47.9758425Z parm:           NVreg_ExcludedGpus:charp
2024-06-05T08:37:47.9758928Z parm:           NVreg_DmaRemapPeerMmio:int
2024-06-05T08:37:47.9759459Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-06-05T08:37:47.9759994Z parm:           NVreg_ImexChannelCount:int
2024-06-05T08:37:47.9760491Z parm:           rm_firmware_active:charp
2024-06-05T08:37:47.9760949Z + HAS_NVIDIA_DRIVER=0
2024-06-05T08:37:47.9761360Z ++ command -v nvidia-smi
2024-06-05T08:37:47.9761791Z + '[' -x /usr/bin/nvidia-smi ']'
2024-06-05T08:37:47.9762315Z + set +e
2024-06-05T08:37:47.9762865Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-06-05T08:37:50.0287377Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-06-05T08:37:50.0288124Z + NVIDIA_SMI_STATUS=0
2024-06-05T08:37:50.0288663Z + '[' 0 -ne 0 ']'
2024-06-05T08:37:50.0289066Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-06-05T08:37:50.0289497Z + HAS_NVIDIA_DRIVER=1
2024-06-05T08:37:50.0290811Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-06-05T08:37:50.0291594Z + set -e
2024-06-05T08:37:50.0291967Z + '[' 1 -eq 0 ']'
2024-06-05T08:37:50.0292594Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-06-05T08:37:50.0293332Z + post_install_nvidia_driver_common
2024-06-05T08:37:50.0293781Z + sudo modprobe nvidia
2024-06-05T08:37:50.0389069Z + echo 'After installing NVIDIA driver'
2024-06-05T08:37:50.0389708Z + lspci
2024-06-05T08:37:50.0390332Z After installing NVIDIA driver
2024-06-05T08:37:50.0473715Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-06-05T08:37:50.0474852Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-06-05T08:37:50.0476370Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-06-05T08:37:50.0477230Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-06-05T08:37:50.0478050Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061
2024-06-05T08:37:50.0478857Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-06-05T08:37:50.0479641Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1)
2024-06-05T08:37:50.0480477Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-06-05T08:37:50.0481108Z + lsmod
2024-06-05T08:37:50.0488156Z Module                  Size  Used by
2024-06-05T08:37:50.0489088Z ib_core               266240  0
2024-06-05T08:37:50.0489663Z xt_nat                 16384  0
2024-06-05T08:37:50.0490223Z nvidia_modeset       1339392  0
2024-06-05T08:37:50.0490770Z veth                   16384  0
2024-06-05T08:37:50.0491345Z nvidia_uvm           4599808  0
2024-06-05T08:37:50.0492045Z nvidia              53989376  2 nvidia_uvm,nvidia_modeset
2024-06-05T08:37:50.0492754Z drm                   421888  1 nvidia
2024-06-05T08:37:50.0493366Z i2c_core               77824  2 nvidia,drm
2024-06-05T08:37:50.0493955Z backlight              16384  1 nvidia_modeset
2024-06-05T08:37:50.0494437Z xt_conntrack           16384  1
2024-06-05T08:37:50.0494853Z ipt_MASQUERADE         16384  1
2024-06-05T08:37:50.0495324Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2024-06-05T08:37:50.0495947Z nf_conntrack_netlink    49152  0
2024-06-05T08:37:50.0498474Z nfnetlink              16384  2 nf_conntrack_netlink
2024-06-05T08:37:50.0498994Z xfrm_user              45056  1
2024-06-05T08:37:50.0499417Z xfrm_algo              16384  1 xfrm_user
2024-06-05T08:37:50.0499873Z iptable_nat            16384  1
2024-06-05T08:37:50.0500291Z nf_conntrack_ipv4      16384  3
2024-06-05T08:37:50.0500756Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2024-06-05T08:37:50.0501275Z nf_nat_ipv4            16384  1 iptable_nat
2024-06-05T08:37:50.0501856Z nf_nat                 32768  3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat
2024-06-05T08:37:50.0502906Z nf_conntrack          155648  8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink
2024-06-05T08:37:50.0503825Z xt_addrtype            16384  2
2024-06-05T08:37:50.0504230Z iptable_filter         16384  1
2024-06-05T08:37:50.0504642Z br_netfilter           24576  0
2024-06-05T08:37:50.0505085Z bridge                172032  1 br_netfilter
2024-06-05T08:37:50.0505560Z stp                    16384  1 bridge
2024-06-05T08:37:50.0506099Z llc                    16384  2 bridge,stp
2024-06-05T08:37:50.0506550Z overlay                86016  0
2024-06-05T08:37:50.0507002Z sunrpc                393216  1
2024-06-05T08:37:50.0507397Z dm_mirror              28672  0
2024-06-05T08:37:50.0507824Z dm_region_hash         20480  1 dm_mirror
2024-06-05T08:37:50.0508351Z dm_log                 20480  2 dm_region_hash,dm_mirror
2024-06-05T08:37:50.0508897Z dm_mod                143360  2 dm_log,dm_mirror
2024-06-05T08:37:50.0509369Z dax                    69632  1 dm_mod
2024-06-05T08:37:50.0509815Z crc32_pclmul           16384  0
2024-06-05T08:37:50.0510375Z ghash_clmulni_intel    16384  0
2024-06-05T08:37:50.0510806Z pcbc                   16384  0
2024-06-05T08:37:50.0511198Z aesni_intel           188416  0
2024-06-05T08:37:50.0511627Z aes_x86_64             20480  1 aesni_intel
2024-06-05T08:37:50.0512111Z crypto_simd            16384  1 aesni_intel
2024-06-05T08:37:50.0512598Z glue_helper            16384  1 aesni_intel
2024-06-05T08:37:50.0513189Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2024-06-05T08:37:50.0513766Z mousedev               24576  0
2024-06-05T08:37:50.0514170Z evdev                  20480  3
2024-06-05T08:37:50.0514559Z psmouse                32768  0
2024-06-05T08:37:50.0514956Z button                 16384  0
2024-06-05T08:37:50.0515437Z ena                   139264  0
2024-06-05T08:37:50.0515841Z ptp                    20480  1 ena
2024-06-05T08:37:50.0516260Z pps_core               20480  1 ptp
2024-06-05T08:37:50.0516680Z crc32c_intel           24576  0
2024-06-05T08:37:50.0517078Z autofs4                49152  2
2024-06-05T08:37:50.0517472Z + modinfo nvidia
2024-06-05T08:37:50.0518182Z filename:       /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2024-06-05T08:37:50.0518922Z alias:          char-major-195-*
2024-06-05T08:37:50.0519335Z version:        550.54.15
2024-06-05T08:37:50.0519710Z supported:      external
2024-06-05T08:37:50.0520070Z license:        NVIDIA
2024-06-05T08:37:50.0520480Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-06-05T08:37:50.0521000Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-06-05T08:37:50.0521541Z srcversion:     833721318DA517F0C2FEC97
2024-06-05T08:37:50.0522034Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-06-05T08:37:50.0522717Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-06-05T08:37:50.0523249Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-06-05T08:37:50.0523767Z depends:        i2c-core,drm
2024-06-05T08:37:50.0524152Z retpoline:      Y
2024-06-05T08:37:50.0524497Z name:           nvidia
2024-06-05T08:37:50.0525093Z vermagic:       4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 
2024-06-05T08:37:50.0525785Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-06-05T08:37:50.0526753Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-06-05T08:37:50.0527407Z parm:           NVreg_ResmanDebugLevel:int
2024-06-05T08:37:50.0527893Z parm:           NVreg_RmLogonRC:int
2024-06-05T08:37:50.0528359Z parm:           NVreg_ModifyDeviceFiles:int
2024-06-05T08:37:50.0528853Z parm:           NVreg_DeviceFileUID:int
2024-06-05T08:37:50.0529329Z parm:           NVreg_DeviceFileGID:int
2024-06-05T08:37:50.0529800Z parm:           NVreg_DeviceFileMode:int
2024-06-05T08:37:50.0530362Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-06-05T08:37:50.0530973Z parm:           NVreg_UsePageAttributeTable:int
2024-06-05T08:37:50.0531536Z parm:           NVreg_EnablePCIeGen3:int
2024-06-05T08:37:50.0532000Z parm:           NVreg_EnableMSI:int
2024-06-05T08:37:50.0532450Z parm:           NVreg_TCEBypassMode:int
2024-06-05T08:37:50.0532946Z parm:           NVreg_EnableStreamMemOPs:int
2024-06-05T08:37:50.0533517Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-06-05T08:37:50.0534140Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-06-05T08:37:50.0534735Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-06-05T08:37:50.0535385Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-06-05T08:37:50.0536026Z parm:           NVreg_DynamicPowerManagement:int
2024-06-05T08:37:50.0536681Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-06-05T08:37:50.0537311Z parm:           NVreg_EnableGpuFirmware:int
2024-06-05T08:37:50.0537850Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-06-05T08:37:50.0538425Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-06-05T08:37:50.0539011Z parm:           NVreg_EnableUserNUMAManagement:int
2024-06-05T08:37:50.0539539Z parm:           NVreg_MemoryPoolSize:int
2024-06-05T08:37:50.0540137Z parm:           NVreg_KMallocHeapMaxSize:int
2024-06-05T08:37:50.0540668Z parm:           NVreg_VMallocHeapMaxSize:int
2024-06-05T08:37:50.0541172Z parm:           NVreg_IgnoreMMIOCheck:int
2024-06-05T08:37:50.0541653Z parm:           NVreg_NvLinkDisable:int
2024-06-05T08:37:50.0542200Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-06-05T08:37:50.0542763Z parm:           NVreg_RegisterPCIDriver:int
2024-06-05T08:37:50.0543272Z parm:           NVreg_EnableResizableBar:int
2024-06-05T08:37:50.0543796Z parm:           NVreg_EnableDbgBreakpoint:int
2024-06-05T08:37:50.0544334Z parm:           NVreg_EnableNonblockingOpen:int
2024-06-05T08:37:50.0544855Z parm:           NVreg_RegistryDwords:charp
2024-06-05T08:37:50.0545465Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-06-05T08:37:50.0545981Z parm:           NVreg_RmMsg:charp
2024-06-05T08:37:50.0546426Z parm:           NVreg_GpuBlacklist:charp
2024-06-05T08:37:50.0546931Z parm:           NVreg_TemporaryFilePath:charp
2024-06-05T08:37:50.0547437Z parm:           NVreg_ExcludedGpus:charp
2024-06-05T08:37:50.0547928Z parm:           NVreg_DmaRemapPeerMmio:int
2024-06-05T08:37:50.0548444Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-06-05T08:37:50.0548955Z parm:           NVreg_ImexChannelCount:int
2024-06-05T08:37:50.0549436Z parm:           rm_firmware_active:charp
2024-06-05T08:37:50.0549865Z + set +e
2024-06-05T08:37:50.0550192Z + nvidia-smi
2024-06-05T08:37:51.6065703Z Wed Jun  5 08:37:51 2024       
2024-06-05T08:37:51.6066673Z +-----------------------------------------------------------------------------------------+
2024-06-05T08:37:51.6067549Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-06-05T08:37:51.6068391Z |-----------------------------------------+------------------------+----------------------+
2024-06-05T08:37:51.6069235Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-06-05T08:37:51.6070191Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-06-05T08:37:51.6070930Z |                                         |                        |               MIG M. |
2024-06-05T08:37:51.6071562Z |=========================================+========================+======================|
2024-06-05T08:37:51.6203273Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-06-05T08:37:51.6204317Z |  0%   34C    P0             58W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-06-05T08:37:51.6205183Z |                                         |                        |                  N/A |
2024-06-05T08:37:51.6206061Z +-----------------------------------------+------------------------+----------------------+
2024-06-05T08:37:51.6206977Z                                                                                          
2024-06-05T08:37:51.6207689Z +-----------------------------------------------------------------------------------------+
2024-06-05T08:37:51.6208365Z | Processes:                                                                              |
2024-06-05T08:37:51.6209119Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-06-05T08:37:51.6209863Z |        ID   ID                                                               Usage      |
2024-06-05T08:37:51.6210476Z |=========================================================================================|
2024-06-05T08:37:51.6211178Z |  No running processes found                                                             |
2024-06-05T08:37:51.6211981Z +-----------------------------------------------------------------------------------------+
2024-06-05T08:37:52.2553822Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-06-05T08:37:53.8126597Z NVIDIA A10G
2024-06-05T08:37:54.2553481Z + NVIDIA_SMI_STATUS=0
2024-06-05T08:37:54.2554195Z + '[' 0 -eq 0 ']'
2024-06-05T08:37:54.2554971Z + echo 'INFO: Ignoring allowed status 0'
2024-06-05T08:37:54.2555469Z + set -e
2024-06-05T08:37:54.2555804Z INFO: Ignoring allowed status 0
2024-06-05T08:37:54.2556883Z == Installing nvidia container toolkit for amzn2 ==
2024-06-05T08:37:54.2558867Z + sudo yum install -y yum-utils
2024-06-05T08:37:54.5587500Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-06-05T08:37:56.0508680Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2024-06-05T08:37:56.0509422Z Nothing to do
2024-06-05T08:37:56.2016186Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2024-06-05T08:37:56.5342111Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-06-05T08:37:56.5560703Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2024-06-05T08:37:56.5561983Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2024-06-05T08:37:56.5563123Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2024-06-05T08:37:56.5677940Z + sudo yum install -y nvidia-docker2
2024-06-05T08:37:56.8832935Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-06-05T08:37:58.3431517Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version
2024-06-05T08:37:58.3432231Z Nothing to do
2024-06-05T08:37:58.4930306Z + sudo systemctl restart docker
2024-06-05T08:38:50.6342054Z Command completed after 1 attempt(s).
2024-06-05T08:38:50.6400679Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-06-05T08:38:50.6401496Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-06-05T08:38:50.6402398Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-06-05T08:38:50.6403105Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-06-05T08:38:50.6411220Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:38:50.6411748Z env:
2024-06-05T08:38:50.6412061Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:38:50.6412559Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:38:50.6413072Z ##[endgroup]
2024-06-05T08:38:50.8632689Z Defaulting to user installation because normal site-packages is not writeable
2024-06-05T08:38:50.8791321Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2024-06-05T08:38:50.8892381Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84)
2024-06-05T08:38:50.9999975Z Prepare all required actions
2024-06-05T08:38:51.0000520Z Getting action download info
2024-06-05T08:38:51.1060974Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-06-05T08:38:51.2679591Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2024-06-05T08:38:51.3735429Z ##[group]Run ./.github/actions/download-build-artifacts
2024-06-05T08:38:51.3735964Z with:
2024-06-05T08:38:51.3736333Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:38:51.3736840Z   s3-bucket: gha-artifacts
2024-06-05T08:38:51.3737222Z env:
2024-06-05T08:38:51.3737537Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:38:51.3738034Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:38:51.3738575Z ##[endgroup]
2024-06-05T08:38:51.3773430Z ##[group]Run seemethere/download-artifact-s3@v4
2024-06-05T08:38:51.3773917Z with:
2024-06-05T08:38:51.3774351Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:38:51.3774849Z   s3-bucket: gha-artifacts
2024-06-05T08:38:51.3775252Z   region: us-east-1
2024-06-05T08:38:51.3775636Z env:
2024-06-05T08:38:51.3775979Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:38:51.3776476Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:38:51.3777012Z ##[endgroup]
2024-06-05T08:38:51.8000044Z (node:775) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-06-05T08:38:51.8000806Z 
2024-06-05T08:38:51.8001090Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-06-05T08:38:51.8002064Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-06-05T08:38:51.8003460Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-06-05T08:38:51.8799540Z Found 1 objects with prefix pytorch/pytorch/9378671038/linux-focal-cuda12.4-py3.10-gcc9-sm86/
2024-06-05T08:38:51.8800685Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-06-05T08:38:59.3781524Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-06-05T08:38:59.3787916Z Artifact download has finished successfully
2024-06-05T08:38:59.3939926Z ##[group]Run unzip -o artifacts.zip
2024-06-05T08:38:59.3940400Z unzip -o artifacts.zip
2024-06-05T08:38:59.3948223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:38:59.3948759Z env:
2024-06-05T08:38:59.3949068Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:38:59.3949566Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:38:59.3950084Z ##[endgroup]
2024-06-05T08:38:59.3980875Z Archive:  artifacts.zip
2024-06-05T08:38:59.3982641Z    creating: dist/
2024-06-05T08:39:01.3771417Z   inflating: dist/torch-2.4.0a0+gitdffed71-cp310-cp310-linux_x86_64.whl  
2024-06-05T08:39:01.3772341Z    creating: build/custom_test_artifacts/
2024-06-05T08:39:01.3772994Z    creating: build/custom_test_artifacts/custom-op-build/
2024-06-05T08:39:01.3773779Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-06-05T08:39:01.3774678Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-06-05T08:39:01.3778827Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-06-05T08:39:01.3779831Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-06-05T08:39:01.3780837Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-06-05T08:39:01.3781887Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-06-05T08:39:01.3782929Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-06-05T08:39:01.3784176Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-06-05T08:39:01.3785576Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-06-05T08:39:01.3786641Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-06-05T08:39:01.3787713Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-06-05T08:39:01.3788952Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-06-05T08:39:01.3790193Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-06-05T08:39:01.3791414Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-06-05T08:39:01.3792619Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-06-05T08:39:01.3793853Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-06-05T08:39:01.3795099Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-06-05T08:39:01.3796205Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-06-05T08:39:01.3797285Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-06-05T08:39:01.3835168Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-06-05T08:39:01.3874896Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-06-05T08:39:01.3876448Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-06-05T08:39:01.3920621Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-06-05T08:39:01.3922260Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-06-05T08:39:01.3923774Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-06-05T08:39:01.3925498Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-06-05T08:39:01.3927214Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-06-05T08:39:01.3928670Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-06-05T08:39:01.3930135Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-06-05T08:39:01.3931592Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-06-05T08:39:01.3933014Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-06-05T08:39:01.3934362Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-06-05T08:39:01.3935676Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-06-05T08:39:01.3936971Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-06-05T08:39:01.3938280Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-06-05T08:39:01.3939548Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-06-05T08:39:01.3940829Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-06-05T08:39:01.4003153Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-06-05T08:39:01.4074922Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-06-05T08:39:01.4076205Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-06-05T08:39:01.4077253Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-06-05T08:39:01.4078170Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-06-05T08:39:01.4079127Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-06-05T08:39:01.4080110Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-06-05T08:39:01.4081190Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-06-05T08:39:01.4082541Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-06-05T08:39:01.4083736Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-06-05T08:39:01.4084829Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-06-05T08:39:01.4085961Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-06-05T08:39:01.4087305Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-06-05T08:39:01.4088454Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-06-05T08:39:01.4089596Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-06-05T08:39:01.4090724Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-06-05T08:39:01.4100001Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-06-05T08:39:01.4230579Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-06-05T08:39:01.4232697Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-06-05T08:39:01.4233984Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-06-05T08:39:01.4235257Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-06-05T08:39:01.4236485Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-06-05T08:39:01.4237636Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-06-05T08:39:01.4238829Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-06-05T08:39:01.4240040Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-06-05T08:39:01.4241247Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-06-05T08:39:01.4242590Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-06-05T08:39:01.4243770Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-06-05T08:39:01.4253537Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-06-05T08:39:01.4333599Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-06-05T08:39:01.4334966Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-06-05T08:39:01.4336106Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-06-05T08:39:01.4337267Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-06-05T08:39:01.4338248Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-06-05T08:39:01.4339211Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-06-05T08:39:01.4340173Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-06-05T08:39:01.4341068Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-06-05T08:39:01.4341924Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-06-05T08:39:01.4342808Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-06-05T08:39:01.4448848Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-06-05T08:39:01.4509598Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-06-05T08:39:01.4510486Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-06-05T08:39:01.4511236Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-06-05T08:39:01.4512098Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-06-05T08:39:01.4518203Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-06-05T08:39:01.4520151Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-06-05T08:39:01.4522340Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-06-05T08:39:01.4523443Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-06-05T08:39:01.4524463Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-06-05T08:39:01.4525630Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-06-05T08:39:01.4527013Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-06-05T08:39:01.4528076Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-06-05T08:39:01.4529121Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-06-05T08:39:01.4530471Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-06-05T08:39:01.4531695Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-06-05T08:39:01.4532947Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-06-05T08:39:01.4534147Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-06-05T08:39:01.4535365Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-06-05T08:39:01.4536579Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-06-05T08:39:01.4537666Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-06-05T08:39:01.4538724Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-06-05T08:39:01.4571554Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-06-05T08:39:01.4611011Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-06-05T08:39:01.4612669Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-06-05T08:39:01.4656434Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-06-05T08:39:01.4657934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-06-05T08:39:01.4659577Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-06-05T08:39:01.4661128Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-06-05T08:39:01.4662614Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-06-05T08:39:01.4664065Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-06-05T08:39:01.4665516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-06-05T08:39:01.4666966Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-06-05T08:39:01.4668384Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-06-05T08:39:01.4669729Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-06-05T08:39:01.4671033Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-06-05T08:39:01.4672302Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-06-05T08:39:01.4673594Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-06-05T08:39:01.4674845Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-06-05T08:39:01.4676117Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-06-05T08:39:01.4738580Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-06-05T08:39:01.4811303Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-06-05T08:39:01.4813142Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-06-05T08:39:01.4814318Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-06-05T08:39:01.4815227Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-06-05T08:39:01.4816166Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-06-05T08:39:01.4817161Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-06-05T08:39:01.4818283Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-06-05T08:39:01.4819535Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-06-05T08:39:01.4820751Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-06-05T08:39:01.4821905Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-06-05T08:39:01.4823100Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-06-05T08:39:01.4824266Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-06-05T08:39:01.4825439Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-06-05T08:39:01.4826607Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-06-05T08:39:01.4827762Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-06-05T08:39:01.4836681Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-06-05T08:39:01.4898812Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-06-05T08:39:01.4900075Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-06-05T08:39:01.4901222Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-06-05T08:39:01.4902244Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-06-05T08:39:01.4903218Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-06-05T08:39:01.4904166Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-06-05T08:39:01.4905131Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-06-05T08:39:01.4906030Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-06-05T08:39:01.4906856Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-06-05T08:39:01.4907722Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-06-05T08:39:01.4954475Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-06-05T08:39:01.4956195Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-06-05T08:39:01.4957774Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-06-05T08:39:01.4959606Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-06-05T08:39:01.4962387Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-06-05T08:39:01.4963452Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-06-05T08:39:01.4985365Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-06-05T08:39:01.4986793Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-06-05T08:39:01.4987900Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-06-05T08:39:01.4989162Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-06-05T08:39:01.4990745Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-06-05T08:39:01.4991871Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-06-05T08:39:01.4992981Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-06-05T08:39:01.4994252Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-06-05T08:39:01.4995541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-06-05T08:39:01.4996795Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-06-05T08:39:01.4998067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-06-05T08:39:01.4999348Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-06-05T08:39:01.5000638Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-06-05T08:39:01.5001900Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-06-05T08:39:01.5003025Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-06-05T08:39:01.5014528Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-06-05T08:39:01.5053987Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-06-05T08:39:01.5055799Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-06-05T08:39:01.5099170Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-06-05T08:39:01.5100743Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-06-05T08:39:01.5102331Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-06-05T08:39:01.5103952Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-06-05T08:39:01.5105517Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-06-05T08:39:01.5107101Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-06-05T08:39:01.5108858Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-06-05T08:39:01.5110400Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-06-05T08:39:01.5111892Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-06-05T08:39:01.5113308Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-06-05T08:39:01.5114681Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-06-05T08:39:01.5116030Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-06-05T08:39:01.5117401Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-06-05T08:39:01.5118734Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-06-05T08:39:01.5120223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-06-05T08:39:01.5179700Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-06-05T08:39:01.5252024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-06-05T08:39:01.5253381Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-06-05T08:39:01.5254511Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-06-05T08:39:01.5255501Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-06-05T08:39:01.5256535Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-06-05T08:39:01.5257605Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-06-05T08:39:01.5258820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-06-05T08:39:01.5260166Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-06-05T08:39:01.5261481Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-06-05T08:39:01.5262699Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-06-05T08:39:01.5263948Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-06-05T08:39:01.5265321Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-06-05T08:39:01.5266592Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-06-05T08:39:01.5267866Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-06-05T08:39:01.5269121Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-06-05T08:39:01.5270438Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-06-05T08:39:01.5381830Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-06-05T08:39:01.5383091Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-06-05T08:39:01.5384353Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-06-05T08:39:01.5385763Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-06-05T08:39:01.5387105Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-06-05T08:39:01.5388388Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-06-05T08:39:01.5389714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-06-05T08:39:01.5391031Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-06-05T08:39:01.5392360Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-06-05T08:39:01.5393676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-06-05T08:39:01.5394990Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-06-05T08:39:01.5405598Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-06-05T08:39:01.5458823Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-06-05T08:39:01.5460201Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-06-05T08:39:01.5461421Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-06-05T08:39:01.5462581Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-06-05T08:39:01.5463611Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-06-05T08:39:01.5464649Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-06-05T08:39:01.5465673Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-06-05T08:39:01.5466647Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-06-05T08:39:01.5467552Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-06-05T08:39:01.5468477Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-06-05T08:39:01.5564873Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-06-05T08:39:01.5606583Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-06-05T08:39:01.5607283Z    creating: build/lib/
2024-06-05T08:39:01.5692680Z   inflating: build/lib/libprotobuf-lite.a  
2024-06-05T08:39:01.6133985Z   inflating: build/lib/libprotobuf.a  
2024-06-05T08:39:01.6143185Z   inflating: build/lib/libpthreadpool.a  
2024-06-05T08:39:01.6150785Z   inflating: build/lib/libcpuinfo.a  
2024-06-05T08:39:01.6158224Z   inflating: build/lib/libcpuinfo_internals.a  
2024-06-05T08:39:01.6158763Z   inflating: build/lib/libclog.a     
2024-06-05T08:39:01.6176750Z   inflating: build/lib/libnnpack.a   
2024-06-05T08:39:01.6178626Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-06-05T08:39:01.6241478Z   inflating: build/lib/libgtest.a    
2024-06-05T08:39:01.6314012Z   inflating: build/lib/libbenchmark.a  
2024-06-05T08:39:01.6375249Z   inflating: build/lib/libasmjit.a   
2024-06-05T08:39:01.6382365Z   inflating: build/lib/libittnotify.a  
2024-06-05T08:39:01.6409813Z   inflating: build/lib/libtensorpipe_uv.a  
2024-06-05T08:39:01.6532506Z   inflating: build/lib/libgloo.a     
2024-06-05T08:39:01.6553529Z   inflating: build/lib/libfmt.a      
2024-06-05T08:39:01.6642143Z   inflating: build/lib/libc10.so     
2024-06-05T08:39:01.6643376Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-06-05T08:39:01.6644456Z   inflating: build/lib/libfoxi_loader.a  
2024-06-05T08:39:01.6645282Z   inflating: build/lib/libtorch_global_deps.so  
2024-06-05T08:39:01.6664178Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-06-05T08:39:01.6681329Z   inflating: build/lib/libgmock.a    
2024-06-05T08:39:01.6682287Z   inflating: build/lib/libgtest_main.a  
2024-06-05T08:39:01.6683863Z   inflating: build/lib/libbenchmark_main.a  
2024-06-05T08:39:01.7170539Z   inflating: build/lib/libprotoc.a   
2024-06-05T08:39:01.7542269Z   inflating: build/lib/libgloo_cuda.a  
2024-06-05T08:39:02.7327103Z   inflating: build/lib/libdnnl.a     
2024-06-05T08:39:02.7880128Z   inflating: build/lib/libtensorpipe.a  
2024-06-05T08:39:02.7932460Z   inflating: build/lib/libc10_cuda.so  
2024-06-05T08:39:02.7933043Z   inflating: build/lib/libgmock_main.a  
2024-06-05T08:39:02.9142431Z   inflating: build/lib/libfbgemm.a   
2024-06-05T08:39:02.9389629Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-06-05T08:39:02.9889167Z   inflating: build/lib/libkineto.a   
2024-06-05T08:39:03.0067950Z   inflating: build/lib/libXNNPACK.a  
2024-06-05T08:39:03.0108824Z   inflating: build/lib/libonnx_proto.a  
2024-06-05T08:39:03.0787119Z   inflating: build/lib/libonnx.a     
2024-06-05T08:39:05.4154540Z   inflating: build/lib/libtorch_cpu.so  
2024-06-05T08:39:05.4158504Z   inflating: build/lib/libunbox_lib.a  
2024-06-05T08:39:05.4162466Z   inflating: build/lib/libshm.so     
2024-06-05T08:39:07.4050402Z   inflating: build/lib/libtorch_cuda.so  
2024-06-05T08:39:07.4051333Z   inflating: build/lib/libtorch.so   
2024-06-05T08:39:07.4053390Z   inflating: build/lib/libc10d_cuda_test.so  
2024-06-05T08:39:08.2613830Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-06-05T08:39:08.4478306Z   inflating: build/lib/libtorch_python.so  
2024-06-05T08:39:08.4543525Z   inflating: build/lib/libtorchbind_test.so  
2024-06-05T08:39:08.4563530Z   inflating: build/lib/libjitbackend_test.so  
2024-06-05T08:39:08.4587913Z   inflating: build/lib/libbackend_with_compiler.so  
2024-06-05T08:39:08.4621521Z   inflating: build/lib/libnnapi_backend.so  
2024-06-05T08:39:08.4622130Z    creating: build/bin/
2024-06-05T08:39:08.4670491Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-06-05T08:39:08.4720488Z   inflating: build/bin/c10_DeviceGuard_test  
2024-06-05T08:39:08.4769847Z   inflating: build/bin/c10_Device_test  
2024-06-05T08:39:08.4827737Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-06-05T08:39:08.4879918Z   inflating: build/bin/c10_Scalar_test  
2024-06-05T08:39:08.4927742Z   inflating: build/bin/c10_StreamGuard_test  
2024-06-05T08:39:08.4977157Z   inflating: build/bin/c10_SymInt_test  
2024-06-05T08:39:08.5030928Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-06-05T08:39:08.5084328Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-06-05T08:39:08.5139021Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-06-05T08:39:08.5207984Z   inflating: build/bin/c10_cow_test  
2024-06-05T08:39:08.5259656Z   inflating: build/bin/c10_Bitset_test  
2024-06-05T08:39:08.5307388Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-06-05T08:39:08.5356958Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-06-05T08:39:08.5405322Z   inflating: build/bin/c10_Half_test  
2024-06-05T08:39:08.5460056Z   inflating: build/bin/c10_LeftRight_test  
2024-06-05T08:39:08.5513444Z   inflating: build/bin/c10_Metaprogramming_test  
2024-06-05T08:39:08.5562149Z   inflating: build/bin/c10_Synchronized_test  
2024-06-05T08:39:08.5616307Z   inflating: build/bin/c10_ThreadLocal_test  
2024-06-05T08:39:08.5666482Z   inflating: build/bin/c10_TypeIndex_test  
2024-06-05T08:39:08.5716239Z   inflating: build/bin/c10_TypeList_test  
2024-06-05T08:39:08.5764160Z   inflating: build/bin/c10_TypeTraits_test  
2024-06-05T08:39:08.5814515Z   inflating: build/bin/c10_accumulate_test  
2024-06-05T08:39:08.5868562Z   inflating: build/bin/c10_bfloat16_test  
2024-06-05T08:39:08.5918310Z   inflating: build/bin/c10_bit_cast_test  
2024-06-05T08:39:08.5972888Z   inflating: build/bin/c10_complex_math_test  
2024-06-05T08:39:08.6028361Z   inflating: build/bin/c10_complex_test  
2024-06-05T08:39:08.6079636Z   inflating: build/bin/c10_exception_test  
2024-06-05T08:39:08.6128294Z   inflating: build/bin/c10_flags_test  
2024-06-05T08:39:08.6177222Z   inflating: build/bin/c10_generic_math_test  
2024-06-05T08:39:08.6336324Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-06-05T08:39:08.6385675Z   inflating: build/bin/c10_irange_test  
2024-06-05T08:39:08.6437581Z   inflating: build/bin/c10_lazy_test  
2024-06-05T08:39:08.6493778Z   inflating: build/bin/c10_logging_test  
2024-06-05T08:39:08.6566029Z   inflating: build/bin/c10_optional_test  
2024-06-05T08:39:08.6626797Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-06-05T08:39:08.6679890Z   inflating: build/bin/c10_registry_test  
2024-06-05T08:39:08.6826794Z   inflating: build/bin/c10_small_vector_test  
2024-06-05T08:39:08.6876749Z   inflating: build/bin/c10_ssize_test  
2024-06-05T08:39:08.6926842Z   inflating: build/bin/c10_string_util_test  
2024-06-05T08:39:08.6984233Z   inflating: build/bin/c10_string_view_test  
2024-06-05T08:39:08.7033839Z   inflating: build/bin/c10_tempfile_test  
2024-06-05T08:39:08.7087834Z   inflating: build/bin/c10_typeid_test  
2024-06-05T08:39:08.7135034Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-06-05T08:39:08.7570857Z   inflating: build/bin/protoc-3.13.0.0  
2024-06-05T08:39:08.8007472Z   inflating: build/bin/protoc        
2024-06-05T08:39:08.8058624Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-06-05T08:39:08.8110969Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-06-05T08:39:08.8162440Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-06-05T08:39:08.8212038Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-06-05T08:39:08.8263258Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-06-05T08:39:08.8314958Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-06-05T08:39:08.8362690Z   inflating: build/bin/c10_cuda_CUDATest  
2024-06-05T08:39:08.8414545Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-06-05T08:39:08.8734259Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-06-05T08:39:08.9069701Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-06-05T08:39:08.9416881Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-06-05T08:39:08.9468527Z   inflating: build/bin/FileStoreTest  
2024-06-05T08:39:08.9523588Z   inflating: build/bin/TCPStoreTest  
2024-06-05T08:39:08.9574988Z   inflating: build/bin/HashStoreTest  
2024-06-05T08:39:08.9587920Z   inflating: build/bin/ProcessGroupMPITest  
2024-06-05T08:39:08.9641028Z   inflating: build/bin/test_edge_op_registration  
2024-06-05T08:39:08.9644600Z   inflating: build/bin/torch_shm_manager  
2024-06-05T08:39:08.9646877Z   inflating: build/bin/example_allreduce  
2024-06-05T08:39:08.9700576Z   inflating: build/bin/test_dist_autograd  
2024-06-05T08:39:08.9768841Z   inflating: build/bin/test_cpp_rpc  
2024-06-05T08:39:08.9771002Z   inflating: build/bin/parallel_benchmark  
2024-06-05T08:39:08.9836223Z   inflating: build/bin/test_mobile_nnc  
2024-06-05T08:39:08.9845006Z   inflating: build/bin/aot_model_compiler_test  
2024-06-05T08:39:09.0179641Z   inflating: build/bin/test_lazy     
2024-06-05T08:39:09.1330395Z   inflating: build/bin/test_api      
2024-06-05T08:39:09.1402357Z   inflating: build/bin/Dict_test     
2024-06-05T08:39:09.1453832Z   inflating: build/bin/Dimname_test  
2024-06-05T08:39:09.1516711Z   inflating: build/bin/MaybeOwned_test  
2024-06-05T08:39:09.1571931Z   inflating: build/bin/NamedTensor_test  
2024-06-05T08:39:09.1630238Z   inflating: build/bin/apply_utils_test  
2024-06-05T08:39:09.1686942Z   inflating: build/bin/atest         
2024-06-05T08:39:09.1748761Z   inflating: build/bin/basic         
2024-06-05T08:39:09.1803257Z   inflating: build/bin/broadcast_test  
2024-06-05T08:39:09.1852046Z   inflating: build/bin/cpu_allocator_test  
2024-06-05T08:39:09.1909441Z   inflating: build/bin/cpu_generator_test  
2024-06-05T08:39:09.1960301Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-06-05T08:39:09.2051257Z   inflating: build/bin/cpu_rng_test  
2024-06-05T08:39:09.2099969Z   inflating: build/bin/dispatch_key_set_test  
2024-06-05T08:39:09.2148947Z   inflating: build/bin/dlconvertor_test  
2024-06-05T08:39:09.2205817Z   inflating: build/bin/extension_backend_test  
2024-06-05T08:39:09.2259778Z   inflating: build/bin/half_test     
2024-06-05T08:39:09.2353160Z   inflating: build/bin/ivalue_test   
2024-06-05T08:39:09.2401704Z   inflating: build/bin/lazy_tensor_test  
2024-06-05T08:39:09.2455231Z   inflating: build/bin/math_kernel_test  
2024-06-05T08:39:09.2508530Z   inflating: build/bin/memory_format_test  
2024-06-05T08:39:09.2560064Z   inflating: build/bin/memory_overlapping_test  
2024-06-05T08:39:09.2611898Z   inflating: build/bin/mobile_memory_cleanup  
2024-06-05T08:39:09.2666390Z   inflating: build/bin/native_test   
2024-06-05T08:39:09.2715510Z   inflating: build/bin/operator_name_test  
2024-06-05T08:39:09.2766601Z   inflating: build/bin/operators_test  
2024-06-05T08:39:09.2816915Z   inflating: build/bin/packedtensoraccessor_test  
2024-06-05T08:39:09.2883331Z   inflating: build/bin/pow_test      
2024-06-05T08:39:09.2938309Z   inflating: build/bin/quantized_test  
2024-06-05T08:39:09.2986980Z   inflating: build/bin/reduce_ops_test  
2024-06-05T08:39:09.3036924Z   inflating: build/bin/reportMemoryUsage_test  
2024-06-05T08:39:09.3092205Z   inflating: build/bin/scalar_tensor_test  
2024-06-05T08:39:09.3149205Z   inflating: build/bin/scalar_test   
2024-06-05T08:39:09.3199689Z   inflating: build/bin/StorageUtils_test  
2024-06-05T08:39:09.3251118Z   inflating: build/bin/stride_properties_test  
2024-06-05T08:39:09.3328552Z   inflating: build/bin/tensor_iterator_test  
2024-06-05T08:39:09.3380315Z   inflating: build/bin/test_parallel  
2024-06-05T08:39:09.3383014Z   inflating: build/bin/thread_init_test  
2024-06-05T08:39:09.3438209Z   inflating: build/bin/type_ptr_test  
2024-06-05T08:39:09.3495857Z   inflating: build/bin/type_test     
2024-06-05T08:39:09.3546785Z   inflating: build/bin/undefined_tensor_test  
2024-06-05T08:39:09.3547798Z   inflating: build/bin/verify_api_visibility  
2024-06-05T08:39:09.3615850Z   inflating: build/bin/legacy_vmap_test  
2024-06-05T08:39:09.3667261Z   inflating: build/bin/weakref_test  
2024-06-05T08:39:09.3716506Z   inflating: build/bin/wrapdim_test  
2024-06-05T08:39:09.3767256Z   inflating: build/bin/xla_tensor_test  
2024-06-05T08:39:09.3826077Z   inflating: build/bin/IListRef_test  
2024-06-05T08:39:09.3928812Z   inflating: build/bin/List_test     
2024-06-05T08:39:09.3994288Z   inflating: build/bin/KernelFunction_test  
2024-06-05T08:39:09.4112851Z   inflating: build/bin/kernel_function_legacy_test  
2024-06-05T08:39:09.4206199Z   inflating: build/bin/kernel_function_test  
2024-06-05T08:39:09.4331562Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-06-05T08:39:09.4432127Z   inflating: build/bin/kernel_lambda_test  
2024-06-05T08:39:09.4491868Z   inflating: build/bin/kernel_stackbased_test  
2024-06-05T08:39:09.4585687Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-06-05T08:39:09.4635241Z   inflating: build/bin/CppSignature_test  
2024-06-05T08:39:09.4682893Z   inflating: build/bin/op_allowlist_test  
2024-06-05T08:39:09.4737347Z   inflating: build/bin/backend_fallback_test  
2024-06-05T08:39:09.4799778Z   inflating: build/bin/inline_container_test  
2024-06-05T08:39:09.5094058Z   inflating: build/bin/op_registration_test  
2024-06-05T08:39:09.5146324Z   inflating: build/bin/cuda_apply_test  
2024-06-05T08:39:09.5195596Z   inflating: build/bin/cuda_allocator_test  
2024-06-05T08:39:09.5248426Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-06-05T08:39:09.5306963Z   inflating: build/bin/cuda_atomic_ops_test  
2024-06-05T08:39:09.5374345Z   inflating: build/bin/cuda_complex_math_test  
2024-06-05T08:39:09.5432027Z   inflating: build/bin/cuda_complex_test  
2024-06-05T08:39:09.5480251Z   inflating: build/bin/cuda_device_test  
2024-06-05T08:39:09.5537400Z   inflating: build/bin/cuda_cub_test  
2024-06-05T08:39:09.5586857Z   inflating: build/bin/cuda_dlconvertor_test  
2024-06-05T08:39:09.5649895Z   inflating: build/bin/cuda_distributions_test  
2024-06-05T08:39:09.5705361Z   inflating: build/bin/cuda_generator_test  
2024-06-05T08:39:09.5754005Z   inflating: build/bin/cuda_half_test  
2024-06-05T08:39:09.5804532Z   inflating: build/bin/cuda_integer_divider_test  
2024-06-05T08:39:09.5852825Z   inflating: build/bin/cuda_optional_test  
2024-06-05T08:39:09.5903481Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-06-05T08:39:09.5955092Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-06-05T08:39:09.6003538Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-06-05T08:39:09.6062727Z   inflating: build/bin/cuda_stream_test  
2024-06-05T08:39:09.6110853Z   inflating: build/bin/cuda_cudnn_test  
2024-06-05T08:39:09.6161994Z   inflating: build/bin/cuda_vectorized_test  
2024-06-05T08:39:09.6176697Z   inflating: build/bin/tutorial_tensorexpr  
2024-06-05T08:39:09.6241179Z   inflating: build/bin/ProcessGroupGlooTest  
2024-06-05T08:39:09.6296542Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-06-05T08:39:09.6357612Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-06-05T08:39:09.6418527Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-06-05T08:39:09.7229871Z   inflating: build/bin/test_tensorexpr  
2024-06-05T08:39:09.7791402Z   inflating: build/bin/test_jit      
2024-06-05T08:39:09.7791961Z    creating: .additional_ci_files/
2024-06-05T08:39:09.7839637Z   inflating: .additional_ci_files/test-times.json  
2024-06-05T08:39:09.8027078Z   inflating: .additional_ci_files/test-class-times.json  
2024-06-05T08:39:09.8056715Z ##[group]Run rm artifacts.zip
2024-06-05T08:39:09.8057140Z rm artifacts.zip
2024-06-05T08:39:09.8064961Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:09.8065506Z env:
2024-06-05T08:39:09.8065841Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:09.8066338Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:09.8066868Z ##[endgroup]
2024-06-05T08:39:09.8768486Z ##[group]Run df -H
2024-06-05T08:39:09.8768840Z df -H
2024-06-05T08:39:09.8776561Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:09.8777104Z env:
2024-06-05T08:39:09.8777425Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:09.8777919Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:09.8778449Z ##[endgroup]
2024-06-05T08:39:09.8804587Z Filesystem      Size  Used Avail Use% Mounted on
2024-06-05T08:39:09.8805294Z devtmpfs         34G     0   34G   0% /dev
2024-06-05T08:39:09.8806148Z tmpfs            34G  5.6M   34G   1% /dev/shm
2024-06-05T08:39:09.8807147Z tmpfs            34G  455k   34G   1% /run
2024-06-05T08:39:09.8807755Z tmpfs            34G     0   34G   0% /sys/fs/cgroup
2024-06-05T08:39:09.8808272Z /dev/nvme0n1p1  162G   86G   77G  53% /
2024-06-05T08:39:09.8847637Z Prepare all required actions
2024-06-05T08:39:09.8848109Z Getting action download info
2024-06-05T08:39:10.0204618Z ##[group]Run ./.github/actions/download-td-artifacts
2024-06-05T08:39:10.0205148Z with:
2024-06-05T08:39:10.0205433Z env:
2024-06-05T08:39:10.0205737Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.0206238Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.0206920Z ##[endgroup]
2024-06-05T08:39:10.0239832Z ##[group]Run seemethere/download-artifact-s3@v4
2024-06-05T08:39:10.0240317Z with:
2024-06-05T08:39:10.0240611Z   name: td_results
2024-06-05T08:39:10.0240966Z   s3-bucket: gha-artifacts
2024-06-05T08:39:10.0241348Z   region: us-east-1
2024-06-05T08:39:10.0241679Z env:
2024-06-05T08:39:10.0242072Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.0242566Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.0243090Z ##[endgroup]
2024-06-05T08:39:10.4335392Z (node:797) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-06-05T08:39:10.4336116Z 
2024-06-05T08:39:10.4336403Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-06-05T08:39:10.4337190Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-06-05T08:39:10.4338083Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-06-05T08:39:10.5063611Z Found 0 objects with prefix pytorch/pytorch/9378671038/td_results/
2024-06-05T08:39:10.5070625Z Artifact download has finished successfully
2024-06-05T08:39:10.5214740Z ##[group]Run mkdir -p .additional_ci_files
2024-06-05T08:39:10.5215524Z mkdir -p .additional_ci_files
2024-06-05T08:39:10.5216415Z mv td_results.json .additional_ci_files/td_results.json
2024-06-05T08:39:10.5227102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:10.5227887Z env:
2024-06-05T08:39:10.5228345Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.5229093Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.5229882Z ##[endgroup]
2024-06-05T08:39:10.5269685Z mv: cannot stat 'td_results.json': No such file or directory
2024-06-05T08:39:10.5294457Z ##[error]Process completed with exit code 1.
2024-06-05T08:39:10.5327560Z ##[group]Run .github/scripts/parse_ref.py
2024-06-05T08:39:10.5328074Z .github/scripts/parse_ref.py
2024-06-05T08:39:10.5335162Z shell: /usr/bin/bash -e {0}
2024-06-05T08:39:10.5335539Z env:
2024-06-05T08:39:10.5335845Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.5336347Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.5336872Z ##[endgroup]
2024-06-05T08:39:10.5568060Z Prepare all required actions
2024-06-05T08:39:10.5606351Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-06-05T08:39:10.5607090Z with:
2024-06-05T08:39:10.5607671Z   github-token: ***
2024-06-05T08:39:10.5608012Z env:
2024-06-05T08:39:10.5608313Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.5608806Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.5609327Z ##[endgroup]
2024-06-05T08:39:10.5625196Z ##[group]Run set -eux
2024-06-05T08:39:10.5625573Z set -eux
2024-06-05T08:39:10.5626233Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-06-05T08:39:10.5634048Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:10.5634589Z env:
2024-06-05T08:39:10.5634906Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:10.5635400Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:10.5636084Z   GITHUB_TOKEN: ***
2024-06-05T08:39:10.5636421Z ##[endgroup]
2024-06-05T08:39:10.5656405Z + python3 .github/scripts/get_workflow_job_id.py 9378671038 i-07bc38970f73de3fc
2024-06-05T08:39:12.5274166Z setting job-id=25823378532
2024-06-05T08:39:12.5275119Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:12.5473563Z Prepare all required actions
2024-06-05T08:39:12.5474050Z Getting action download info
2024-06-05T08:39:12.6792050Z ##[group]Run ./.github/actions/filter-test-configs
2024-06-05T08:39:12.6792543Z with:
2024-06-05T08:39:12.6793039Z   github-token: ***
2024-06-05T08:39:12.6801201Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-06-05T08:39:12.6810403Z   job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:12.6811156Z env:
2024-06-05T08:39:12.6811467Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:12.6811959Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:12.6812468Z ##[endgroup]
2024-06-05T08:39:12.6854745Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-06-05T08:39:12.6855341Z with:
2024-06-05T08:39:12.6855644Z   shell: bash
2024-06-05T08:39:12.6855980Z   timeout_minutes: 10
2024-06-05T08:39:12.6856343Z   max_attempts: 5
2024-06-05T08:39:12.6856686Z   retry_wait_seconds: 30
2024-06-05T08:39:12.6857882Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-06-05T08:39:12.6859131Z   polling_interval_seconds: 1
2024-06-05T08:39:12.6859544Z   warning_on_retry: true
2024-06-05T08:39:12.6859933Z   continue_on_error: false
2024-06-05T08:39:12.6860306Z env:
2024-06-05T08:39:12.6860609Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:12.6861095Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:12.6861765Z   GITHUB_TOKEN: ***
2024-06-05T08:39:12.6862111Z ##[endgroup]
2024-06-05T08:39:12.7358983Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-06-05T08:39:12.9513292Z Defaulting to user installation because normal site-packages is not writeable
2024-06-05T08:39:12.9665957Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1)
2024-06-05T08:39:12.9811397Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1)
2024-06-05T08:39:12.9817989Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2)
2024-06-05T08:39:12.9826672Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.18)
2024-06-05T08:39:13.0025012Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7)
2024-06-05T08:39:13.0037838Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12)
2024-06-05T08:39:13.7360762Z Command completed after 1 attempt(s).
2024-06-05T08:39:13.7406885Z ##[group]Run set -x
2024-06-05T08:39:13.7407258Z set -x
2024-06-05T08:39:13.7407590Z 
2024-06-05T08:39:13.7408199Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-06-05T08:39:13.7408924Z # in runner workspace
2024-06-05T08:39:13.7409501Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-06-05T08:39:13.7417388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:13.7417948Z env:
2024-06-05T08:39:13.7418255Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:13.7418755Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:13.7419279Z ##[endgroup]
2024-06-05T08:39:13.7441138Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-06-05T08:39:13.7654256Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-06-05T08:39:13.7654859Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-06-05T08:39:13.7655362Z echo "Job name: ${JOB_NAME}"
2024-06-05T08:39:13.7655801Z 
2024-06-05T08:39:13.7656404Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-06-05T08:39:13.7657150Z # in runner workspace
2024-06-05T08:39:13.7657776Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-06-05T08:39:13.7658469Z   --workflow "${GITHUB_WORKFLOW}" \
2024-06-05T08:39:13.7659129Z   --job-name "${JOB_NAME}" \
2024-06-05T08:39:13.7667826Z   --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-06-05T08:39:13.7676762Z   --selected-test-configs "" \
2024-06-05T08:39:13.7677258Z   --pr-number "${PR_NUMBER}" \
2024-06-05T08:39:13.7677720Z   --tag "${TAG}" \
2024-06-05T08:39:13.7678150Z   --event-name "${EVENT_NAME}" \
2024-06-05T08:39:13.7678645Z   --schedule "${SCHEDULE}" \
2024-06-05T08:39:13.7679119Z   --branch "${HEAD_BRANCH}"
2024-06-05T08:39:13.7687144Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:13.7687679Z env:
2024-06-05T08:39:13.7687991Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:13.7688499Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:13.7689217Z   GITHUB_TOKEN: ***
2024-06-05T08:39:13.7689913Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:13.7690670Z   PR_NUMBER: 
2024-06-05T08:39:13.7691020Z   TAG: ciflow/inductor/127669
2024-06-05T08:39:13.7691429Z   EVENT_NAME: push
2024-06-05T08:39:13.7691763Z   SCHEDULE: 
2024-06-05T08:39:13.7692085Z   HEAD_BRANCH: 
2024-06-05T08:39:13.7692415Z ##[endgroup]
2024-06-05T08:39:13.7711803Z Workflow: inductor
2024-06-05T08:39:13.7712726Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:13.9986847Z INFO:root:Found no test-config label on the PR, so all test configs are included
2024-06-05T08:39:14.0962770Z ##[group]Run echo "Filtered matrix:"
2024-06-05T08:39:14.0963286Z echo "Filtered matrix:"
2024-06-05T08:39:14.0971939Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu", "unstable": "unstable"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-06-05T08:39:14.0980745Z 
2024-06-05T08:39:14.0981065Z echo
2024-06-05T08:39:14.0981500Z echo "Is the current job unstable? False"
2024-06-05T08:39:14.0982086Z 
2024-06-05T08:39:14.0982404Z echo
2024-06-05T08:39:14.0982816Z echo "Is keep-going label set? False"
2024-06-05T08:39:14.0983299Z 
2024-06-05T08:39:14.0983613Z echo
2024-06-05T08:39:14.0983985Z echo "Renabled issues? "
2024-06-05T08:39:14.0991713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:14.0992255Z env:
2024-06-05T08:39:14.0992581Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:14.0993087Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:14.0993623Z ##[endgroup]
2024-06-05T08:39:14.1013821Z Filtered matrix:
2024-06-05T08:39:14.1024585Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu, unstable: unstable}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu, unstable: unstable}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-06-05T08:39:14.1033200Z 
2024-06-05T08:39:14.1033369Z Is the current job unstable? False
2024-06-05T08:39:14.1033683Z 
2024-06-05T08:39:14.1033959Z Is keep-going label set? False
2024-06-05T08:39:14.1034244Z 
2024-06-05T08:39:14.1034380Z Renabled issues? 
2024-06-05T08:39:14.1077856Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-06-05T08:39:14.1078616Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-06-05T08:39:14.1086631Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T08:39:14.1087164Z env:
2024-06-05T08:39:14.1087474Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:14.1087967Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:14.1088489Z   JOB_TIMEOUT: 240
2024-06-05T08:39:14.1088817Z ##[endgroup]
2024-06-05T08:39:14.1159503Z ##[group]Run set -x
2024-06-05T08:39:14.1159938Z set -x
2024-06-05T08:39:14.1160270Z 
2024-06-05T08:39:14.1160660Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-06-05T08:39:14.1161242Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-06-05T08:39:14.1161848Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-06-05T08:39:14.1162546Z   TEST_COMMAND=.ci/onnx/test.sh
2024-06-05T08:39:14.1162996Z else
2024-06-05T08:39:14.1163366Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-06-05T08:39:14.1163833Z fi
2024-06-05T08:39:14.1164136Z 
2024-06-05T08:39:14.1164642Z # detached container should get cleaned up by teardown_ec2_linux
2024-06-05T08:39:14.1165451Z # TODO: Stop building test binaries as part of the build phase
2024-06-05T08:39:14.1166182Z # Used for GPU_FLAG since that doesn't play nice
2024-06-05T08:39:14.1167031Z # shellcheck disable=SC2086,SC2090
2024-06-05T08:39:14.1167530Z container_name=$(docker run \
2024-06-05T08:39:14.1167992Z   ${GPU_FLAG:-} \
2024-06-05T08:39:14.1168399Z   -e BUILD_ENVIRONMENT \
2024-06-05T08:39:14.1168836Z   -e PR_NUMBER \
2024-06-05T08:39:14.1169225Z   -e GITHUB_ACTIONS \
2024-06-05T08:39:14.1169651Z   -e GITHUB_REPOSITORY \
2024-06-05T08:39:14.1170092Z   -e GITHUB_WORKFLOW \
2024-06-05T08:39:14.1170511Z   -e GITHUB_JOB \
2024-06-05T08:39:14.1170898Z   -e GITHUB_RUN_ID \
2024-06-05T08:39:14.1171315Z   -e GITHUB_RUN_NUMBER \
2024-06-05T08:39:14.1171758Z   -e GITHUB_RUN_ATTEMPT \
2024-06-05T08:39:14.1172181Z   -e JOB_ID \
2024-06-05T08:39:14.1172550Z   -e JOB_NAME \
2024-06-05T08:39:14.1172930Z   -e BASE_SHA \
2024-06-05T08:39:14.1173307Z   -e BRANCH \
2024-06-05T08:39:14.1173665Z   -e SHA1 \
2024-06-05T08:39:14.1174045Z   -e AWS_DEFAULT_REGION \
2024-06-05T08:39:14.1174483Z   -e IN_WHEEL_TEST \
2024-06-05T08:39:14.1174895Z   -e SHARD_NUMBER \
2024-06-05T08:39:14.1175294Z   -e TEST_CONFIG \
2024-06-05T08:39:14.1175697Z   -e NUM_TEST_SHARDS \
2024-06-05T08:39:14.1176125Z   -e REENABLED_ISSUES \
2024-06-05T08:39:14.1176574Z   -e CONTINUE_THROUGH_ERROR \
2024-06-05T08:39:14.1177031Z   -e VERBOSE_TEST_LOGS \
2024-06-05T08:39:14.1177463Z   -e NO_TEST_TIMEOUT \
2024-06-05T08:39:14.1177873Z   -e NO_TD \
2024-06-05T08:39:14.1178238Z   -e TD_DISTRIBUTED \
2024-06-05T08:39:14.1178650Z   -e PR_LABELS \
2024-06-05T08:39:14.1179089Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-06-05T08:39:14.1179583Z   -e SCCACHE_BUCKET \
2024-06-05T08:39:14.1180007Z   -e SCCACHE_S3_KEY_PREFIX \
2024-06-05T08:39:14.1180451Z   -e XLA_CUDA \
2024-06-05T08:39:14.1180890Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-06-05T08:39:14.1181439Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-06-05T08:39:14.1182124Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-06-05T08:39:14.1182677Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-06-05T08:39:14.1183188Z   -e HUGGING_FACE_HUB_TOKEN \
2024-06-05T08:39:14.1183643Z   -e DASHBOARD_TAG \
2024-06-05T08:39:14.1184153Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-06-05T08:39:14.1184753Z   --security-opt seccomp=unconfined \
2024-06-05T08:39:14.1185255Z   --cap-add=SYS_PTRACE \
2024-06-05T08:39:14.1185677Z   --ipc=host \
2024-06-05T08:39:14.1186062Z   --shm-size="${SHM_SIZE}" \
2024-06-05T08:39:14.1186493Z   --tty \
2024-06-05T08:39:14.1186836Z   --detach \
2024-06-05T08:39:14.1187224Z   --name="${container_name}" \
2024-06-05T08:39:14.1187673Z   --user jenkins \
2024-06-05T08:39:14.1188272Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-06-05T08:39:14.1188882Z   -w /var/lib/jenkins/workspace \
2024-06-05T08:39:14.1189364Z   "${DOCKER_IMAGE}"
2024-06-05T08:39:14.1189736Z )
2024-06-05T08:39:14.1190176Z # Propagate download.pytorch.org IP to container
2024-06-05T08:39:14.1191161Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-06-05T08:39:14.1192194Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-06-05T08:39:14.1193163Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-06-05T08:39:14.1200359Z shell: /usr/bin/bash -e {0}
2024-06-05T08:39:14.1200738Z env:
2024-06-05T08:39:14.1201043Z   GIT_DEFAULT_BRANCH: main
2024-06-05T08:39:14.1201529Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:14.1202307Z   BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:39:14.1202840Z   PR_NUMBER: 
2024-06-05T08:39:14.1203207Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-06-05T08:39:14.1203651Z   GITHUB_WORKFLOW: inductor
2024-06-05T08:39:14.1204034Z   GITHUB_JOB: test
2024-06-05T08:39:14.1204386Z   GITHUB_RUN_ID: 9378671038
2024-06-05T08:39:14.1204773Z   GITHUB_RUN_NUMBER: 72689
2024-06-05T08:39:14.1205148Z   GITHUB_RUN_ATTEMPT: 1
2024-06-05T08:39:14.1205518Z   JOB_ID: 25823378532
2024-06-05T08:39:14.1206213Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:14.1207161Z   BRANCH: 
2024-06-05T08:39:14.1207529Z   SHA1: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:14.1208086Z   BASE_SHA: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:14.1208609Z   TEST_CONFIG: inductor_torchbench
2024-06-05T08:39:14.1209034Z   SHARD_NUMBER: 2
2024-06-05T08:39:14.1209368Z   NUM_TEST_SHARDS: 2
2024-06-05T08:39:14.1209722Z   REENABLED_ISSUES: 
2024-06-05T08:39:14.1210108Z   CONTINUE_THROUGH_ERROR: False
2024-06-05T08:39:14.1210526Z   VERBOSE_TEST_LOGS: False
2024-06-05T08:39:14.1210923Z   NO_TEST_TIMEOUT: False
2024-06-05T08:39:14.1211290Z   NO_TD: False
2024-06-05T08:39:14.1211622Z   TD_DISTRIBUTED: False
2024-06-05T08:39:14.1212067Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-06-05T08:39:14.1212594Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-06-05T08:39:14.1213013Z   SHM_SIZE: 2g
2024-06-05T08:39:14.1214153Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:39:14.1215376Z   XLA_CUDA: 
2024-06-05T08:39:14.1215890Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-06-05T08:39:14.1216542Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-06-05T08:39:14.1217017Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-06-05T08:39:14.1217455Z   DASHBOARD_TAG: 
2024-06-05T08:39:14.1218002Z   HUGGING_FACE_HUB_TOKEN: ***
2024-06-05T08:39:14.1218401Z ##[endgroup]
2024-06-05T08:39:14.1238114Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]]
2024-06-05T08:39:14.1238968Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]]
2024-06-05T08:39:14.1239526Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-06-05T08:39:14.1244960Z +++ nproc --ignore=2
2024-06-05T08:39:14.1257785Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9378671038 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T08:39:25.5194772Z + container_name=b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T08:39:25.5197077Z + docker exec -i b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e sudo bash -c '/bin/cat >> /etc/hosts'
2024-06-05T08:39:25.5198026Z + grep download.pytorch.org /etc/hosts
2024-06-05T08:39:25.5892120Z + echo DOCKER_CONTAINER_ID=b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T08:39:25.5894473Z ++ echo dist/torch-2.4.0a0+gitdffed71-cp310-cp310-linux_x86_64.whl
2024-06-05T08:39:25.5896649Z + docker exec -t b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e sh -c 'pip install dist/torch-2.4.0a0+gitdffed71-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-06-05T08:39:25.9808963Z Processing ./dist/torch-2.4.0a0+gitdffed71-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitdffed71)
2024-06-05T08:39:26.2949079Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (3.13.1)
2024-06-05T08:39:26.2951863Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (4.12.1)
2024-06-05T08:39:26.2954603Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (1.12.1)
2024-06-05T08:39:26.2957024Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (2.8.8)
2024-06-05T08:39:26.2959828Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (3.1.4)
2024-06-05T08:39:26.2962748Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (2024.2.0)
2024-06-05T08:39:26.2988885Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (3.3.0)
2024-06-05T08:39:26.3047701Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (1.21.2)
2024-06-05T08:39:26.3480697Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (2.1.5)
2024-06-05T08:39:26.3638733Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitdffed71->torch==2.4.0a0+gitdffed71) (1.3.0)
2024-06-05T08:39:27.6816458Z Installing collected packages: torch
2024-06-05T08:39:36.3161078Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-06-05T08:39:36.3163317Z timm 0.9.7 requires torchvision, which is not installed.
2024-06-05T08:39:36.3164036Z Successfully installed torch-2.4.0a0+gitdffed71
2024-06-05T08:39:36.3924167Z ++ dirname .ci/pytorch/test.sh
2024-06-05T08:39:36.3929031Z + source .ci/pytorch/common.sh
2024-06-05T08:39:36.3931860Z +++ dirname .ci/pytorch/common.sh
2024-06-05T08:39:36.3938167Z ++ source .ci/pytorch/common_utils.sh
2024-06-05T08:39:36.3939936Z +++ declare -f -t trap_add
2024-06-05T08:39:36.3945650Z ++ set -ex
2024-06-05T08:39:36.3946379Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-06-05T08:39:36.3946943Z ++ BUILD_TEST_LIBTORCH=0
2024-06-05T08:39:36.3947481Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]]
2024-06-05T08:39:36.3949360Z ++ stat -c %u /var/lib/jenkins/workspace
2024-06-05T08:39:36.3957015Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-06-05T08:39:36.3957557Z + trap_add cleanup_workspace EXIT
2024-06-05T08:39:36.3958015Z + trap_add_cmd=cleanup_workspace
2024-06-05T08:39:36.3958415Z + shift
2024-06-05T08:39:36.3960086Z + for trap_add_name in "$@"
2024-06-05T08:39:36.3963038Z +++ trap -p EXIT
2024-06-05T08:39:36.3963967Z ++ eval 'extract_trap_cmd '
2024-06-05T08:39:36.3964440Z +++ extract_trap_cmd
2024-06-05T08:39:36.3964829Z +++ printf '%s\n' ''
2024-06-05T08:39:36.3965247Z ++ printf '%s\n' cleanup_workspace
2024-06-05T08:39:36.3966202Z + trap -- '
2024-06-05T08:39:36.3966920Z cleanup_workspace' EXIT
2024-06-05T08:39:36.3967441Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-06-05T08:39:36.8695246Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-06-05T08:39:36.8706498Z + echo 'Environment variables:'
2024-06-05T08:39:36.8707017Z Environment variables:
2024-06-05T08:39:36.8707388Z + env
2024-06-05T08:39:36.8711303Z INSTALLED_DB=yes
2024-06-05T08:39:36.8711748Z NV_LIBCUBLAS_VERSION=12.4.2.65-1
2024-06-05T08:39:36.8712297Z NVIDIA_VISIBLE_DEVICES=all
2024-06-05T08:39:36.8712899Z NV_NVML_DEV_VERSION=12.4.99-1
2024-06-05T08:39:36.8713538Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T08:39:36.8714298Z CONTINUE_THROUGH_ERROR=False
2024-06-05T08:39:36.8715041Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4
2024-06-05T08:39:36.8715688Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1
2024-06-05T08:39:36.8716533Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:39:36.8717195Z HOSTNAME=b23c091d19f2
2024-06-05T08:39:36.8718277Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8719149Z GITHUB_ACTION=__self
2024-06-05T08:39:36.8719538Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-06-05T08:39:36.8726271Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-06-05T08:39:36.8733847Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1
2024-06-05T08:39:36.8734400Z NV_NVTX_VERSION=12.4.99-1
2024-06-05T08:39:36.8734838Z GITHUB_RUN_NUMBER=72689
2024-06-05T08:39:36.8735230Z TEST_CONFIG=inductor_torchbench
2024-06-05T08:39:36.8735664Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-06-05T08:39:36.8736178Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-06-05T08:39:36.8736678Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1
2024-06-05T08:39:36.8737157Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1
2024-06-05T08:39:36.8737611Z NV_LIBNPP_VERSION=12.2.5.2-1
2024-06-05T08:39:36.8738290Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-06-05T08:39:36.8738841Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-06-05T08:39:36.8739349Z GITHUB_REF_TYPE=tag
2024-06-05T08:39:36.8739712Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-06-05T08:39:36.8740126Z NCCL_VERSION=2.20.5-1
2024-06-05T08:39:36.8740562Z BASE_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8741035Z XLA_CUDA=
2024-06-05T08:39:36.8741577Z HUGGING_FACE_HUB_TOKEN=***
2024-06-05T08:39:36.8743630Z ***
2024-06-05T08:39:36.8743974Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-06-05T08:39:36.8744413Z GITHUB_REPOSITORY_ID=65600975
2024-06-05T08:39:36.8744820Z GITHUB_ACTIONS=true
2024-06-05T08:39:36.8745193Z NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:36.8745735Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1
2024-06-05T08:39:36.8746294Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1
2024-06-05T08:39:36.8746802Z SHA1=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8747350Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-06-05T08:39:36.8747885Z GITHUB_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8748729Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/127669
2024-06-05T08:39:36.8749505Z UCC_HOME=/usr
2024-06-05T08:39:36.8749889Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1
2024-06-05T08:39:36.8750333Z VERBOSE_TEST_LOGS=False
2024-06-05T08:39:36.8750708Z NVIDIA_PRODUCT_NAME=CUDA
2024-06-05T08:39:36.8751199Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-06-05T08:39:36.8751743Z GITHUB_REF=refs/tags/ciflow/inductor/127669
2024-06-05T08:39:36.8752242Z NV_CUDA_CUDART_VERSION=12.4.99-1
2024-06-05T08:39:36.8752656Z SHARD_NUMBER=2
2024-06-05T08:39:36.8753000Z GITHUB_REF_PROTECTED=false
2024-06-05T08:39:36.8753388Z HOME=/var/lib/jenkins
2024-06-05T08:39:36.8753786Z GITHUB_API_URL=https://api.github.com
2024-06-05T08:39:36.8754268Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-06-05T08:39:36.8754783Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-06-05T08:39:36.8755306Z SCCACHE_S3_KEY_PREFIX=inductor
2024-06-05T08:39:36.8755708Z CUDA_VERSION=12.4.0
2024-06-05T08:39:36.8756169Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1
2024-06-05T08:39:36.8756653Z NUM_TEST_SHARDS=2
2024-06-05T08:39:36.8756985Z UCX_HOME=/usr
2024-06-05T08:39:36.8757533Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1
2024-06-05T08:39:36.8758667Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8759952Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:36.8761231Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8762540Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-06-05T08:39:36.8763248Z GITHUB_EVENT_NAME=push
2024-06-05T08:39:36.8763617Z DASHBOARD_TAG=
2024-06-05T08:39:36.8763956Z GITHUB_RUN_ID=9378671038
2024-06-05T08:39:36.8764446Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1
2024-06-05T08:39:36.8765174Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-06-05T08:39:36.8766320Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8767463Z GITHUB_ACTOR=pytorch-bot[bot]
2024-06-05T08:39:36.8767903Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1
2024-06-05T08:39:36.8768314Z PR_NUMBER=
2024-06-05T08:39:36.8768635Z GITHUB_RUN_ATTEMPT=1
2024-06-05T08:39:36.8769005Z ANACONDA_PYTHON_VERSION=3.10
2024-06-05T08:39:36.8769494Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-06-05T08:39:36.8769994Z TERM=xterm
2024-06-05T08:39:36.8770366Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1
2024-06-05T08:39:36.8770817Z INSTALLED_VISION=yes
2024-06-05T08:39:36.8771165Z BRANCH=
2024-06-05T08:39:36.8771486Z OPENSSL_ROOT_DIR=/opt/openssl
2024-06-05T08:39:36.8771921Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-06-05T08:39:36.8772388Z CUDA_PATH=/usr/local/cuda
2024-06-05T08:39:36.8773270Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-06-05T08:39:36.8774090Z GITHUB_SERVER_URL=https://github.com
2024-06-05T08:39:36.8774614Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-06-05T08:39:36.8775118Z REENABLED_ISSUES=
2024-06-05T08:39:36.8775448Z SHLVL=1
2024-06-05T08:39:36.8775739Z MAX_JOBS=14
2024-06-05T08:39:36.8776082Z NV_CUDA_LIB_VERSION=12.4.0-1
2024-06-05T08:39:36.8776471Z NVARCH=x86_64
2024-06-05T08:39:36.8776805Z GITHUB_ACTOR_ID=54816060
2024-06-05T08:39:36.8777301Z GITHUB_WORKFLOW_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8777873Z GITHUB_REF_NAME=ciflow/inductor/127669
2024-06-05T08:39:36.8778400Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-06-05T08:39:36.8779092Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-06-05T08:39:36.8779693Z GITHUB_JOB=test
2024-06-05T08:39:36.8780126Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4
2024-06-05T08:39:36.8780738Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-06-05T08:39:36.8781300Z NO_TEST_TIMEOUT=False
2024-06-05T08:39:36.8781661Z TD_DISTRIBUTED=False
2024-06-05T08:39:36.8782076Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1
2024-06-05T08:39:36.8782557Z GITHUB_REPOSITORY=pytorch/pytorch
2024-06-05T08:39:36.8783012Z NV_NVPROF_VERSION=12.4.99-1
2024-06-05T08:39:36.8783404Z GITHUB_RETENTION_DAYS=90
2024-06-05T08:39:36.8783790Z OPENSSL_DIR=/opt/openssl
2024-06-05T08:39:36.8784177Z GITHUB_ACTION_REPOSITORY=
2024-06-05T08:39:36.8785271Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-05T08:39:36.8786378Z GITHUB_BASE_REF=
2024-06-05T08:39:36.8786742Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-06-05T08:39:36.8787146Z CI=true
2024-06-05T08:39:36.8787484Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1
2024-06-05T08:39:36.8787943Z GITHUB_REPOSITORY_OWNER=pytorch
2024-06-05T08:39:36.8788360Z JOB_ID=25823378532
2024-06-05T08:39:36.8788708Z INSTALLED_PROTOBUF=yes
2024-06-05T08:39:36.8789072Z GITHUB_HEAD_REF=
2024-06-05T08:39:36.8789409Z GITHUB_ACTION_REF=
2024-06-05T08:39:36.8789877Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-06-05T08:39:36.8790383Z GITHUB_WORKFLOW=inductor
2024-06-05T08:39:36.8790775Z DEBIAN_FRONTEND=noninteractive
2024-06-05T08:39:36.8791727Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8792594Z NO_TD=False
2024-06-05T08:39:36.8792928Z SKIP_SCCACHE_INITIALIZATION=1
2024-06-05T08:39:36.8793332Z _=/usr/bin/env
2024-06-05T08:39:36.8793847Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-06-05T08:39:36.8857412Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-06-05T08:39:36.8858370Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-06-05T08:39:36.8859293Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-06-05T08:39:36.8860250Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-06-05T08:39:36.8861039Z + BUILD_DIR=build
2024-06-05T08:39:36.8861410Z + BUILD_RENAMED_DIR=build_renamed
2024-06-05T08:39:36.8861842Z + BUILD_BIN_DIR=build/bin
2024-06-05T08:39:36.8862224Z + SHARD_NUMBER=2
2024-06-05T08:39:36.8862566Z + NUM_TEST_SHARDS=2
2024-06-05T08:39:36.8862916Z + export VALGRIND=ON
2024-06-05T08:39:36.8863280Z + VALGRIND=ON
2024-06-05T08:39:36.8863796Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]]
2024-06-05T08:39:36.8864335Z + [[ 0 == \1 ]]
2024-06-05T08:39:36.8864659Z + [[ False == \1 ]]
2024-06-05T08:39:36.8865173Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]]
2024-06-05T08:39:36.8865739Z ++ realpath build/custom_test_artifacts
2024-06-05T08:39:36.8870142Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-06-05T08:39:36.8870866Z + [[ -n '' ]]
2024-06-05T08:39:36.8871365Z + echo 'Environment variables'
2024-06-05T08:39:36.8871789Z Environment variables
2024-06-05T08:39:36.8872155Z + env
2024-06-05T08:39:36.8875885Z INSTALLED_DB=yes
2024-06-05T08:39:36.8876465Z NV_LIBCUBLAS_VERSION=12.4.2.65-1
2024-06-05T08:39:36.8877060Z NVIDIA_VISIBLE_DEVICES=all
2024-06-05T08:39:36.8877677Z NV_NVML_DEV_VERSION=12.4.99-1
2024-06-05T08:39:36.8878519Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T08:39:36.8879342Z CONTINUE_THROUGH_ERROR=False
2024-06-05T08:39:36.8880075Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4
2024-06-05T08:39:36.8880809Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1
2024-06-05T08:39:36.8881557Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T08:39:36.8882330Z HOSTNAME=b23c091d19f2
2024-06-05T08:39:36.8883526Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8884593Z GITHUB_ACTION=__self
2024-06-05T08:39:36.8884985Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-06-05T08:39:36.8891101Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-06-05T08:39:36.8897194Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1
2024-06-05T08:39:36.8897742Z NV_NVTX_VERSION=12.4.99-1
2024-06-05T08:39:36.8898130Z GITHUB_RUN_NUMBER=72689
2024-06-05T08:39:36.8898528Z TEST_CONFIG=inductor_torchbench
2024-06-05T08:39:36.8898958Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-06-05T08:39:36.8899465Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-06-05T08:39:36.8899955Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1
2024-06-05T08:39:36.8900433Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1
2024-06-05T08:39:36.8900887Z NV_LIBNPP_VERSION=12.2.5.2-1
2024-06-05T08:39:36.8901362Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-06-05T08:39:36.8901909Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-06-05T08:39:36.8902405Z GITHUB_REF_TYPE=tag
2024-06-05T08:39:36.8902770Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-06-05T08:39:36.8903193Z NCCL_VERSION=2.20.5-1
2024-06-05T08:39:36.8903624Z BASE_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8904251Z XLA_CUDA=
2024-06-05T08:39:36.8904767Z HUGGING_FACE_HUB_TOKEN=***
2024-06-05T08:39:36.8905195Z ***
2024-06-05T08:39:36.8905509Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-06-05T08:39:36.8905944Z GITHUB_REPOSITORY_ID=65600975
2024-06-05T08:39:36.8906349Z GITHUB_ACTIONS=true
2024-06-05T08:39:36.8906770Z NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T08:39:36.8907295Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1
2024-06-05T08:39:36.8907866Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1
2024-06-05T08:39:36.8908366Z SHA1=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8908913Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-06-05T08:39:36.8909440Z GITHUB_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8910288Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/127669
2024-06-05T08:39:36.8911050Z UCC_HOME=/usr
2024-06-05T08:39:36.8911501Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1
2024-06-05T08:39:36.8911955Z VERBOSE_TEST_LOGS=False
2024-06-05T08:39:36.8912337Z NVIDIA_PRODUCT_NAME=CUDA
2024-06-05T08:39:36.8912823Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-06-05T08:39:36.8913361Z GITHUB_REF=refs/tags/ciflow/inductor/127669
2024-06-05T08:39:36.8913866Z NV_CUDA_CUDART_VERSION=12.4.99-1
2024-06-05T08:39:36.8914271Z SHARD_NUMBER=2
2024-06-05T08:39:36.8914622Z GITHUB_REF_PROTECTED=false
2024-06-05T08:39:36.8915009Z HOME=/var/lib/jenkins
2024-06-05T08:39:36.8915409Z GITHUB_API_URL=https://api.github.com
2024-06-05T08:39:36.8915878Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-06-05T08:39:36.8916393Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-06-05T08:39:36.8916915Z SCCACHE_S3_KEY_PREFIX=inductor
2024-06-05T08:39:36.8917322Z CUDA_VERSION=12.4.0
2024-06-05T08:39:36.8917774Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1
2024-06-05T08:39:36.8918259Z NUM_TEST_SHARDS=2
2024-06-05T08:39:36.8918590Z UCX_HOME=/usr
2024-06-05T08:39:36.8919133Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1
2024-06-05T08:39:36.8920272Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8921548Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-06-05T08:39:36.8922990Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8924150Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-06-05T08:39:36.8924842Z GITHUB_EVENT_NAME=push
2024-06-05T08:39:36.8925205Z DASHBOARD_TAG=
2024-06-05T08:39:36.8925539Z GITHUB_RUN_ID=9378671038
2024-06-05T08:39:36.8926026Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1
2024-06-05T08:39:36.8926806Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-06-05T08:39:36.8941657Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8942887Z GITHUB_ACTOR=pytorch-bot[bot]
2024-06-05T08:39:36.8943340Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1
2024-06-05T08:39:36.8943749Z PR_NUMBER=
2024-06-05T08:39:36.8944068Z GITHUB_RUN_ATTEMPT=1
2024-06-05T08:39:36.8944411Z VALGRIND=ON
2024-06-05T08:39:36.8944743Z ANACONDA_PYTHON_VERSION=3.10
2024-06-05T08:39:36.8945228Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-06-05T08:39:36.8945719Z TERM=xterm
2024-06-05T08:39:36.8946088Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1
2024-06-05T08:39:36.8946540Z INSTALLED_VISION=yes
2024-06-05T08:39:36.8946884Z BRANCH=
2024-06-05T08:39:36.8947193Z OPENSSL_ROOT_DIR=/opt/openssl
2024-06-05T08:39:36.8947630Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-06-05T08:39:36.8948091Z CUDA_PATH=/usr/local/cuda
2024-06-05T08:39:36.8948890Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-06-05T08:39:36.8949696Z GITHUB_SERVER_URL=https://github.com
2024-06-05T08:39:36.8950217Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-06-05T08:39:36.8950873Z REENABLED_ISSUES=
2024-06-05T08:39:36.8951197Z SHLVL=1
2024-06-05T08:39:36.8951485Z MAX_JOBS=14
2024-06-05T08:39:36.8951834Z NV_CUDA_LIB_VERSION=12.4.0-1
2024-06-05T08:39:36.8952222Z NVARCH=x86_64
2024-06-05T08:39:36.8952546Z GITHUB_ACTOR_ID=54816060
2024-06-05T08:39:36.8953046Z GITHUB_WORKFLOW_SHA=dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T08:39:36.8953625Z GITHUB_REF_NAME=ciflow/inductor/127669
2024-06-05T08:39:36.8954149Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-06-05T08:39:36.8954841Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-06-05T08:39:36.8955442Z GITHUB_JOB=test
2024-06-05T08:39:36.8955872Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4
2024-06-05T08:39:36.8956465Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-06-05T08:39:36.8957016Z NO_TEST_TIMEOUT=False
2024-06-05T08:39:36.8957381Z TD_DISTRIBUTED=False
2024-06-05T08:39:36.8957878Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1
2024-06-05T08:39:36.8958358Z GITHUB_REPOSITORY=pytorch/pytorch
2024-06-05T08:39:36.8958814Z NV_NVPROF_VERSION=12.4.99-1
2024-06-05T08:39:36.8959208Z GITHUB_RETENTION_DAYS=90
2024-06-05T08:39:36.8959590Z OPENSSL_DIR=/opt/openssl
2024-06-05T08:39:36.8959968Z GITHUB_ACTION_REPOSITORY=
2024-06-05T08:39:36.8961068Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-05T08:39:36.8962290Z GITHUB_BASE_REF=
2024-06-05T08:39:36.8962674Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-06-05T08:39:36.8963104Z CI=true
2024-06-05T08:39:36.8963443Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1
2024-06-05T08:39:36.8963895Z GITHUB_REPOSITORY_OWNER=pytorch
2024-06-05T08:39:36.8964294Z JOB_ID=25823378532
2024-06-05T08:39:36.8964640Z INSTALLED_PROTOBUF=yes
2024-06-05T08:39:36.8964998Z GITHUB_HEAD_REF=
2024-06-05T08:39:36.8965325Z GITHUB_ACTION_REF=
2024-06-05T08:39:36.8965784Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-06-05T08:39:36.8966296Z GITHUB_WORKFLOW=inductor
2024-06-05T08:39:36.8966987Z DEBIAN_FRONTEND=noninteractive
2024-06-05T08:39:36.8968017Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c22b9677-d351-4ce5-88d9-f2f2d3fe4900
2024-06-05T08:39:36.8968942Z NO_TD=False
2024-06-05T08:39:36.8969279Z SKIP_SCCACHE_INITIALIZATION=1
2024-06-05T08:39:36.8969680Z _=/usr/bin/env
2024-06-05T08:39:36.8970046Z + echo 'Testing pytorch'
2024-06-05T08:39:36.8970421Z Testing pytorch
2024-06-05T08:39:36.8970771Z + export LANG=C.UTF-8
2024-06-05T08:39:36.8971136Z + LANG=C.UTF-8
2024-06-05T08:39:36.8971449Z + PR_NUMBER=
2024-06-05T08:39:36.8971826Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]]
2024-06-05T08:39:36.8972379Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-06-05T08:39:36.8972911Z + [[ inductor_torchbench == \s\l\o\w ]]
2024-06-05T08:39:36.8973568Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-06-05T08:39:36.8974295Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]]
2024-06-05T08:39:36.8974883Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-06-05T08:39:36.8975390Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-06-05T08:39:36.8975864Z + [[ inductor_torchbench == *crossref* ]]
2024-06-05T08:39:36.8976465Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-06-05T08:39:36.8977122Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-06-05T08:39:36.8977789Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-06-05T08:39:36.8978379Z + pip_install --user ninja==1.10.2
2024-06-05T08:39:36.8978944Z + pip install --progress-bar off --user ninja==1.10.2
2024-06-05T08:39:37.2547937Z Collecting ninja==1.10.2
2024-06-05T08:39:37.2720441Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-06-05T08:39:37.2860963Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-06-05T08:39:38.5769279Z Installing collected packages: ninja
2024-06-05T08:39:38.5840320Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-06-05T08:39:38.5842863Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-06-05T08:39:38.5872322Z Successfully installed ninja-1.10.2
2024-06-05T08:39:38.6695944Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-05T08:39:38.6698210Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-05T08:39:38.6699826Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-06-05T08:39:38.6700370Z + install_tlparse
2024-06-05T08:39:38.6701075Z + pip_install --user tlparse==0.3.7
2024-06-05T08:39:38.6701681Z + pip install --progress-bar off --user tlparse==0.3.7
2024-06-05T08:39:39.0230665Z Collecting tlparse==0.3.7
2024-06-05T08:39:39.0380580Z   Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes)
2024-06-05T08:39:39.0470760Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-06-05T08:39:40.3640921Z Installing collected packages: tlparse
2024-06-05T08:39:40.4009903Z Successfully installed tlparse-0.3.7
2024-06-05T08:39:40.4846044Z ++ python -m site --user-base
2024-06-05T08:39:40.5000306Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-05T08:39:40.5002219Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]]
2024-06-05T08:39:40.5003010Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]]
2024-06-05T08:39:40.5003785Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-06-05T08:39:40.5004937Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-06-05T08:39:40.5006259Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass
2024-06-05T08:39:40.5007342Z + cd test
2024-06-05T08:39:40.5007969Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-06-05T08:39:41.9858233Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-06-05T08:39:41.9859211Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-06-05T08:39:41.9860187Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-06-05T08:39:41.9860789Z + pushd test
2024-06-05T08:39:41.9861144Z ~/workspace/test ~/workspace
2024-06-05T08:39:41.9861709Z ++ python -c 'import torch; print(torch.version.cuda)'
2024-06-05T08:39:43.4421457Z + CUDA_VERSION=12.4
2024-06-05T08:39:43.4422109Z + '[' 12.4 == 12.4 ']'
2024-06-05T08:39:43.4422485Z + ISCUDA124=cu124
2024-06-05T08:39:43.4422820Z + popd
2024-06-05T08:39:43.4423116Z ~/workspace
2024-06-05T08:39:43.4425549Z + DYNAMO_BENCHMARK_FLAGS=()
2024-06-05T08:39:43.4426021Z + [[ inductor_torchbench == *dynamo_eager* ]]
2024-06-05T08:39:43.4426605Z + [[ inductor_torchbench == *aot_eager* ]]
2024-06-05T08:39:43.4427124Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-06-05T08:39:43.4427636Z + [[ inductor_torchbench == *inductor* ]]
2024-06-05T08:39:43.4428127Z + [[ inductor_torchbench != *perf* ]]
2024-06-05T08:39:43.4428642Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-06-05T08:39:43.4429129Z + [[ inductor_torchbench == *dynamic* ]]
2024-06-05T08:39:43.4429644Z + [[ inductor_torchbench == *cpu_inductor* ]]
2024-06-05T08:39:43.4430201Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-06-05T08:39:43.4453640Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-06-05T08:39:43.4454376Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-06-05T08:39:43.4455913Z + cd test
2024-06-05T08:39:43.4456691Z + python -c 'import torch; print(torch.__config__.show())'
2024-06-05T08:39:44.7581248Z PyTorch built with:
2024-06-05T08:39:44.7581827Z   - GCC 9.4
2024-06-05T08:39:44.7582325Z   - C++ Version: 201703
2024-06-05T08:39:44.7583308Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-06-05T08:39:44.7584485Z   - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-06-05T08:39:44.7585200Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-06-05T08:39:44.7585773Z   - LAPACK is enabled (usually provided by MKL)
2024-06-05T08:39:44.7586298Z   - NNPACK is enabled
2024-06-05T08:39:44.7586711Z   - CPU capability usage: AVX2
2024-06-05T08:39:44.7587157Z   - CUDA Runtime 12.4
2024-06-05T08:39:44.7587719Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-06-05T08:39:44.7588305Z   - CuDNN 90.1
2024-06-05T08:39:44.7588882Z   - Magma 2.6.1
2024-06-05T08:39:44.7596115Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-06-05T08:39:44.7603020Z 
2024-06-05T08:39:45.0311277Z + cd test
2024-06-05T08:39:45.0311995Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-06-05T08:39:46.2224243Z ATen/Parallel:
2024-06-05T08:39:46.2224677Z 	at::get_num_threads() : 8
2024-06-05T08:39:46.2225247Z 	at::get_num_interop_threads() : 16
2024-06-05T08:39:46.2225849Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-06-05T08:39:46.2226301Z 	omp_get_max_threads() : 8
2024-06-05T08:39:46.2227316Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-06-05T08:39:46.2228153Z 	mkl_get_max_threads() : 8
2024-06-05T08:39:46.2228837Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-06-05T08:39:46.2229522Z std::thread::hardware_concurrency() : 16
2024-06-05T08:39:46.2230004Z Environment variables:
2024-06-05T08:39:46.2230383Z 	OMP_NUM_THREADS : [not set]
2024-06-05T08:39:46.2230791Z 	MKL_NUM_THREADS : [not set]
2024-06-05T08:39:46.2231215Z ATen parallel backend: OpenMP
2024-06-05T08:39:46.2231493Z 
2024-06-05T08:39:46.4663999Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-06-05T08:39:46.4664835Z + [[ inductor_torchbench == *backward* ]]
2024-06-05T08:39:46.4665392Z + [[ inductor_torchbench == *xla* ]]
2024-06-05T08:39:46.4665887Z + [[ inductor_torchbench == *executorch* ]]
2024-06-05T08:39:46.4666439Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]]
2024-06-05T08:39:46.4667177Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-06-05T08:39:46.4667882Z + [[ inductor_torchbench == distributed ]]
2024-06-05T08:39:46.4668452Z + [[ inductor_torchbench == deploy ]]
2024-06-05T08:39:46.4669014Z + [[ inductor_torchbench == *inductor_distributed* ]]
2024-06-05T08:39:46.4669694Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]]
2024-06-05T08:39:46.4670572Z + [[ inductor_torchbench == *huggingface* ]]
2024-06-05T08:39:46.4671086Z + [[ inductor_torchbench == *timm* ]]
2024-06-05T08:39:46.4671573Z + [[ inductor_torchbench == *torchbench* ]]
2024-06-05T08:39:46.4672089Z + [[ inductor_torchbench == *cpu_inductor* ]]
2024-06-05T08:39:46.4672579Z + install_torchaudio cuda
2024-06-05T08:39:46.4672957Z + local commit
2024-06-05T08:39:46.4673308Z ++ get_pinned_commit audio
2024-06-05T08:39:46.4673718Z ++ cat .github/ci_commit_pins/audio.txt
2024-06-05T08:39:46.4685511Z + commit=1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:46.4686113Z + [[ cuda == \c\u\d\a ]]
2024-06-05T08:39:46.4686791Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-06-05T08:39:46.4687757Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:46.4689465Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:46.7789206Z Collecting git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:46.7792293Z   Cloning https://github.com/pytorch/audio.git (to revision 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0) to /tmp/pip-req-build-06e0llef
2024-06-05T08:39:46.7807470Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-06e0llef
2024-06-05T08:39:47.5194208Z   Running command git rev-parse -q --verify 'sha^1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0'
2024-06-05T08:39:47.5209603Z   Running command git fetch -q https://github.com/pytorch/audio.git 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:47.9536113Z   Resolved https://github.com/pytorch/audio.git to commit 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0
2024-06-05T08:39:47.9537235Z   Running command git submodule update --init --recursive -q
2024-06-05T08:39:50.0764593Z   Preparing metadata (setup.py) ... [?25l- done
2024-06-05T08:39:50.0795121Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.2.0a0+1980f8a) (2.4.0a0+gitdffed71)
2024-06-05T08:39:50.0857841Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.13.1)
2024-06-05T08:39:50.0863130Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (4.12.1)
2024-06-05T08:39:50.0865526Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (1.12.1)
2024-06-05T08:39:50.0867883Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2.8.8)
2024-06-05T08:39:50.0870759Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.1.4)
2024-06-05T08:39:50.0873433Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2024.2.0)
2024-06-05T08:39:50.1302499Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.2.0a0+1980f8a) (2.1.5)
2024-06-05T08:39:50.1459042Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.2.0a0+1980f8a) (1.3.0)
2024-06-05T08:39:50.1529081Z Building wheels for collected packages: torchaudio
2024-06-05T08:41:58.4102968Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-06-05T08:41:58.4130508Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.2.0a0+1980f8a-cp310-cp310-linux_x86_64.whl size=2310148 sha256=7230f3fe3393deef0be0b890cf0ce09ff77c2cf7ad4f13f14b82cea5dbfd1594
2024-06-05T08:41:58.4132907Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3c/69/04/2f8953929b90f7f98a00671a3b24cfeb70fc70e57328823e5c
2024-06-05T08:41:58.4159193Z Successfully built torchaudio
2024-06-05T08:41:59.6335134Z Installing collected packages: torchaudio
2024-06-05T08:41:59.8244494Z Successfully installed torchaudio-2.2.0a0+1980f8a
2024-06-05T08:41:59.9550576Z + install_torchtext
2024-06-05T08:41:59.9551533Z + local data_commit
2024-06-05T08:41:59.9552397Z + local text_commit
2024-06-05T08:41:59.9554891Z ++ get_pinned_commit data
2024-06-05T08:41:59.9555531Z ++ cat .github/ci_commit_pins/data.txt
2024-06-05T08:41:59.9570114Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:41:59.9572290Z ++ get_pinned_commit text
2024-06-05T08:41:59.9572719Z ++ cat .github/ci_commit_pins/text.txt
2024-06-05T08:41:59.9582477Z + text_commit=b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:41:59.9583582Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:41:59.9585109Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:42:00.2734768Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:42:00.2738876Z   Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-phurrw2l
2024-06-05T08:42:00.2754920Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-phurrw2l
2024-06-05T08:42:00.7611184Z   Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f'
2024-06-05T08:42:00.7626007Z   Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:42:01.0090946Z   Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:42:01.1833674Z   Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-06-05T08:42:01.1834997Z   Running command git submodule update --init --recursive -q
2024-06-05T08:43:09.9236229Z   Preparing metadata (setup.py) ... [?25l- done
2024-06-05T08:43:09.9290003Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.18)
2024-06-05T08:43:09.9292434Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3)
2024-06-05T08:43:09.9298901Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.4.0a0+gitdffed71)
2024-06-05T08:43:09.9362513Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1)
2024-06-05T08:43:09.9368156Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.1)
2024-06-05T08:43:09.9371707Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1)
2024-06-05T08:43:09.9374894Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8)
2024-06-05T08:43:09.9378086Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4)
2024-06-05T08:43:09.9381129Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0)
2024-06-05T08:43:09.9575624Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2)
2024-06-05T08:43:09.9582625Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7)
2024-06-05T08:43:09.9588977Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2)
2024-06-05T08:43:10.0062555Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5)
2024-06-05T08:43:10.0233895Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0)
2024-06-05T08:43:10.0311137Z Building wheels for collected packages: torchdata
2024-06-05T08:43:11.7542780Z   Building wheel for torchdata (setup.py) ... [?25l- \ | done
2024-06-05T08:43:11.7549596Z [?25h  Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=8495804d5a5c451b6a22a93c2def6d4fc11bad6a188fa777a45bfb59f142eb65
2024-06-05T08:43:11.7552292Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce
2024-06-05T08:43:11.7579208Z Successfully built torchdata
2024-06-05T08:43:12.9717493Z Installing collected packages: torchdata
2024-06-05T08:43:13.0562462Z Successfully installed torchdata-0.7.0a0+11bb5b8
2024-06-05T08:43:15.6065850Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:15.6067414Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:15.9195976Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:15.9199850Z   Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-qd0noakn
2024-06-05T08:43:15.9215569Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-qd0noakn
2024-06-05T08:43:16.7659753Z   Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db'
2024-06-05T08:43:16.7678348Z   Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:17.2418812Z   Running command git checkout -q b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:17.4342727Z   Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db
2024-06-05T08:43:17.4344222Z   Running command git submodule update --init --recursive -q
2024-06-05T08:43:24.5281419Z   Preparing metadata (setup.py) ... [?25l- done
2024-06-05T08:43:24.5332478Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4)
2024-06-05T08:43:24.5335567Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3)
2024-06-05T08:43:24.5338468Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.4.0a0+gitdffed71)
2024-06-05T08:43:24.5341134Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2)
2024-06-05T08:43:24.5344045Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8)
2024-06-05T08:43:24.5404618Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2)
2024-06-05T08:43:24.5410033Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7)
2024-06-05T08:43:24.5416544Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.18)
2024-06-05T08:43:24.5421389Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2)
2024-06-05T08:43:24.5477994Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1)
2024-06-05T08:43:24.5483666Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.1)
2024-06-05T08:43:24.5485890Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1)
2024-06-05T08:43:24.5489352Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8)
2024-06-05T08:43:24.5492259Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4)
2024-06-05T08:43:24.5494466Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0)
2024-06-05T08:43:24.6179176Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5)
2024-06-05T08:43:24.6341954Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0)
2024-06-05T08:43:24.6421496Z Building wheels for collected packages: torchtext
2024-06-05T08:44:00.8746386Z   Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-06-05T08:44:00.8773912Z [?25h  Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045300 sha256=0eab557c5c957ae0f5902fb3676e5372c4a814b62879f7a76cf582ae273158dc
2024-06-05T08:44:00.8776693Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf
2024-06-05T08:44:00.8799560Z Successfully built torchtext
2024-06-05T08:44:02.0665031Z Installing collected packages: torchtext
2024-06-05T08:44:02.1649691Z Successfully installed torchtext-0.16.0a0+b0ebddc
2024-06-05T08:44:02.2914370Z + install_torchvision
2024-06-05T08:44:02.2917267Z + local orig_preload
2024-06-05T08:44:02.2918301Z + local commit
2024-06-05T08:44:02.2919238Z ++ get_pinned_commit vision
2024-06-05T08:44:02.2919965Z ++ cat .github/ci_commit_pins/vision.txt
2024-06-05T08:44:02.2931974Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:02.2932462Z + orig_preload=
2024-06-05T08:44:02.2932841Z + '[' -n '' ']'
2024-06-05T08:44:02.2933725Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:02.2935218Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:02.6070975Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:02.6075998Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-so6svrtf
2024-06-05T08:44:02.6095036Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-so6svrtf
2024-06-05T08:44:04.0391291Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-06-05T08:44:04.0407100Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:05.2913997Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:05.5192039Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-06-05T08:44:07.8168715Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-06-05T08:44:07.8217338Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2)
2024-06-05T08:44:07.8220670Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitdffed71)
2024-06-05T08:44:07.8226058Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0)
2024-06-05T08:44:07.8432312Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1)
2024-06-05T08:44:07.8438587Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.1)
2024-06-05T08:44:07.8442164Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1)
2024-06-05T08:44:07.8445297Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-06-05T08:44:07.8448755Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-06-05T08:44:07.8452252Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0)
2024-06-05T08:44:07.8885832Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5)
2024-06-05T08:44:07.9042542Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-06-05T08:44:07.9118687Z Building wheels for collected packages: torchvision
2024-06-05T08:45:18.4615336Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2024-06-05T08:45:18.4642253Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2022270 sha256=ea4361a4a4464ebe6422a7a81e45b25e17864d40810a0d08d6ce89f090023bd6
2024-06-05T08:45:18.4644730Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-06-05T08:45:18.4680143Z Successfully built torchvision
2024-06-05T08:45:19.6753542Z Installing collected packages: torchvision
2024-06-05T08:45:20.0516688Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-06-05T08:45:20.1668650Z + '[' -n '' ']'
2024-06-05T08:45:20.1669031Z + id=1
2024-06-05T08:45:20.1669432Z + pip_install opencv-python==4.8.0.74
2024-06-05T08:45:20.1670064Z + pip install --progress-bar off opencv-python==4.8.0.74
2024-06-05T08:45:20.5979206Z Collecting opencv-python==4.8.0.74
2024-06-05T08:45:20.6142480Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2024-06-05T08:45:20.6329274Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2)
2024-06-05T08:45:20.6392704Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2024-06-05T08:45:22.3890401Z Installing collected packages: opencv-python
2024-06-05T08:45:23.2135070Z Successfully installed opencv-python-4.8.0.74
2024-06-05T08:45:23.3135512Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]]
2024-06-05T08:45:23.3136605Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]]
2024-06-05T08:45:23.3137589Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]]
2024-06-05T08:45:23.3138307Z + checkout_install_torchbench
2024-06-05T08:45:23.3140030Z + local commit
2024-06-05T08:45:23.3140478Z ++ get_pinned_commit torchbench
2024-06-05T08:45:23.3140957Z ++ cat .github/ci_commit_pins/torchbench.txt
2024-06-05T08:45:23.3148039Z + commit=d6015d42d9a1834bc7595c4bd6852562fb80b30b
2024-06-05T08:45:23.3148756Z + git clone https://github.com/pytorch/benchmark torchbench
2024-06-05T08:45:23.3160107Z Cloning into 'torchbench'...
2024-06-05T08:45:23.4914822Z remote: Enumerating objects: 29467, done.
2024-06-05T08:45:23.4918280Z remote: Counting objects:   0% (1/4697)
2024-06-05T08:45:23.4918851Z remote: Counting objects:   1% (47/4697)
2024-06-05T08:45:23.4919806Z remote: Counting objects:   2% (94/4697)
2024-06-05T08:45:23.4920516Z remote: Counting objects:   3% (141/4697)
2024-06-05T08:45:23.4921085Z remote: Counting objects:   4% (188/4697)
2024-06-05T08:45:23.4921647Z remote: Counting objects:   5% (235/4697)
2024-06-05T08:45:23.4923286Z remote: Counting objects:   6% (282/4697)
2024-06-05T08:45:23.4923928Z remote: Counting objects:   7% (329/4697)
2024-06-05T08:45:23.4924532Z remote: Counting objects:   8% (376/4697)
2024-06-05T08:45:23.4925098Z remote: Counting objects:   9% (423/4697)
2024-06-05T08:45:23.4925648Z remote: Counting objects:  10% (470/4697)
2024-06-05T08:45:23.4926317Z remote: Counting objects:  11% (517/4697)
2024-06-05T08:45:23.4927118Z remote: Counting objects:  12% (564/4697)
2024-06-05T08:45:23.4927668Z remote: Counting objects:  13% (611/4697)
2024-06-05T08:45:23.4928297Z remote: Counting objects:  14% (658/4697)
2024-06-05T08:45:23.4928995Z remote: Counting objects:  15% (705/4697)
2024-06-05T08:45:23.4929546Z remote: Counting objects:  16% (752/4697)
2024-06-05T08:45:23.4930150Z remote: Counting objects:  17% (799/4697)
2024-06-05T08:45:23.4930807Z remote: Counting objects:  18% (846/4697)
2024-06-05T08:45:23.4931412Z remote: Counting objects:  19% (893/4697)
2024-06-05T08:45:23.4932048Z remote: Counting objects:  20% (940/4697)
2024-06-05T08:45:23.4932628Z remote: Counting objects:  21% (987/4697)
2024-06-05T08:45:23.4933221Z remote: Counting objects:  22% (1034/4697)
2024-06-05T08:45:23.4933817Z remote: Counting objects:  23% (1081/4697)
2024-06-05T08:45:23.4934399Z remote: Counting objects:  24% (1128/4697)
2024-06-05T08:45:23.4934986Z remote: Counting objects:  25% (1175/4697)
2024-06-05T08:45:23.4935570Z remote: Counting objects:  26% (1222/4697)
2024-06-05T08:45:23.4936229Z remote: Counting objects:  27% (1269/4697)
2024-06-05T08:45:23.4936787Z remote: Counting objects:  28% (1316/4697)
2024-06-05T08:45:23.4937345Z remote: Counting objects:  29% (1363/4697)
2024-06-05T08:45:23.4937903Z remote: Counting objects:  30% (1410/4697)
2024-06-05T08:45:23.4938457Z remote: Counting objects:  31% (1457/4697)
2024-06-05T08:45:23.4939180Z remote: Counting objects:  32% (1504/4697)
2024-06-05T08:45:23.4939867Z remote: Counting objects:  33% (1551/4697)
2024-06-05T08:45:23.4940446Z remote: Counting objects:  34% (1597/4697)
2024-06-05T08:45:23.4941178Z remote: Counting objects:  35% (1644/4697)
2024-06-05T08:45:23.4942195Z remote: Counting objects:  36% (1691/4697)
2024-06-05T08:45:23.4942802Z remote: Counting objects:  37% (1738/4697)
2024-06-05T08:45:23.4943419Z remote: Counting objects:  38% (1785/4697)
2024-06-05T08:45:23.4944183Z remote: Counting objects:  39% (1832/4697)
2024-06-05T08:45:23.4944869Z remote: Counting objects:  40% (1879/4697)
2024-06-05T08:45:23.4945423Z remote: Counting objects:  41% (1926/4697)
2024-06-05T08:45:23.4946015Z remote: Counting objects:  42% (1973/4697)
2024-06-05T08:45:23.4946636Z remote: Counting objects:  43% (2020/4697)
2024-06-05T08:45:23.4947182Z remote: Counting objects:  44% (2067/4697)
2024-06-05T08:45:23.4947733Z remote: Counting objects:  45% (2114/4697)
2024-06-05T08:45:23.4948282Z remote: Counting objects:  46% (2161/4697)
2024-06-05T08:45:23.4948931Z remote: Counting objects:  47% (2208/4697)
2024-06-05T08:45:23.4949648Z remote: Counting objects:  48% (2255/4697)
2024-06-05T08:45:23.4950207Z remote: Counting objects:  49% (2302/4697)
2024-06-05T08:45:23.4950766Z remote: Counting objects:  50% (2349/4697)
2024-06-05T08:45:23.4951418Z remote: Counting objects:  51% (2396/4697)
2024-06-05T08:45:23.4951973Z remote: Counting objects:  52% (2443/4697)
2024-06-05T08:45:23.4952526Z remote: Counting objects:  53% (2490/4697)
2024-06-05T08:45:23.4953077Z remote: Counting objects:  54% (2537/4697)
2024-06-05T08:45:23.4953624Z remote: Counting objects:  55% (2584/4697)
2024-06-05T08:45:23.4954204Z remote: Counting objects:  56% (2631/4697)
2024-06-05T08:45:23.4954906Z remote: Counting objects:  57% (2678/4697)
2024-06-05T08:45:23.4955466Z remote: Counting objects:  58% (2725/4697)
2024-06-05T08:45:23.4956017Z remote: Counting objects:  59% (2772/4697)
2024-06-05T08:45:23.4956560Z remote: Counting objects:  60% (2819/4697)
2024-06-05T08:45:23.4957109Z remote: Counting objects:  61% (2866/4697)
2024-06-05T08:45:23.4957669Z remote: Counting objects:  62% (2913/4697)
2024-06-05T08:45:23.4958224Z remote: Counting objects:  63% (2960/4697)
2024-06-05T08:45:23.4958769Z remote: Counting objects:  64% (3007/4697)
2024-06-05T08:45:23.4959323Z remote: Counting objects:  65% (3054/4697)
2024-06-05T08:45:23.4959874Z remote: Counting objects:  66% (3101/4697)
2024-06-05T08:45:23.4960417Z remote: Counting objects:  67% (3147/4697)
2024-06-05T08:45:23.4960968Z remote: Counting objects:  68% (3194/4697)
2024-06-05T08:45:23.4961518Z remote: Counting objects:  69% (3241/4697)
2024-06-05T08:45:23.4962198Z remote: Counting objects:  70% (3288/4697)
2024-06-05T08:45:23.4962798Z remote: Counting objects:  71% (3335/4697)
2024-06-05T08:45:23.4963346Z remote: Counting objects:  72% (3382/4697)
2024-06-05T08:45:23.4963896Z remote: Counting objects:  73% (3429/4697)
2024-06-05T08:45:23.4964440Z remote: Counting objects:  74% (3476/4697)
2024-06-05T08:45:23.4965052Z remote: Counting objects:  75% (3523/4697)
2024-06-05T08:45:23.4965664Z remote: Counting objects:  76% (3570/4697)
2024-06-05T08:45:23.4966220Z remote: Counting objects:  77% (3617/4697)
2024-06-05T08:45:23.4966911Z remote: Counting objects:  78% (3664/4697)
2024-06-05T08:45:23.4967458Z remote: Counting objects:  79% (3711/4697)
2024-06-05T08:45:23.4968005Z remote: Counting objects:  80% (3758/4697)
2024-06-05T08:45:23.4968550Z remote: Counting objects:  81% (3805/4697)
2024-06-05T08:45:23.4969100Z remote: Counting objects:  82% (3852/4697)
2024-06-05T08:45:23.4969639Z remote: Counting objects:  83% (3899/4697)
2024-06-05T08:45:23.4970185Z remote: Counting objects:  84% (3946/4697)
2024-06-05T08:45:23.4970734Z remote: Counting objects:  85% (3993/4697)
2024-06-05T08:45:23.4971272Z remote: Counting objects:  86% (4040/4697)
2024-06-05T08:45:23.4971817Z remote: Counting objects:  87% (4087/4697)
2024-06-05T08:45:23.4972365Z remote: Counting objects:  88% (4134/4697)
2024-06-05T08:45:23.4972965Z remote: Counting objects:  89% (4181/4697)
2024-06-05T08:45:23.4973503Z remote: Counting objects:  90% (4228/4697)
2024-06-05T08:45:23.4974162Z remote: Counting objects:  91% (4275/4697)
2024-06-05T08:45:23.4974708Z remote: Counting objects:  92% (4322/4697)
2024-06-05T08:45:23.4975252Z remote: Counting objects:  93% (4369/4697)
2024-06-05T08:45:23.4975789Z remote: Counting objects:  94% (4416/4697)
2024-06-05T08:45:23.4976334Z remote: Counting objects:  95% (4463/4697)
2024-06-05T08:45:23.4976878Z remote: Counting objects:  96% (4510/4697)
2024-06-05T08:45:23.4977423Z remote: Counting objects:  97% (4557/4697)
2024-06-05T08:45:23.4977977Z remote: Counting objects:  98% (4604/4697)
2024-06-05T08:45:23.4978524Z remote: Counting objects:  99% (4651/4697)
2024-06-05T08:45:23.4979067Z remote: Counting objects: 100% (4697/4697)
2024-06-05T08:45:23.4979647Z remote: Counting objects: 100% (4697/4697), done.
2024-06-05T08:45:23.4980233Z remote: Compressing objects:   0% (1/944)
2024-06-05T08:45:23.4989610Z remote: Compressing objects:   1% (10/944)
2024-06-05T08:45:23.5022391Z remote: Compressing objects:   2% (19/944)
2024-06-05T08:45:23.5071483Z remote: Compressing objects:   3% (29/944)
2024-06-05T08:45:23.5078580Z remote: Compressing objects:   4% (38/944)
2024-06-05T08:45:23.5101196Z remote: Compressing objects:   5% (48/944)
2024-06-05T08:45:23.5122589Z remote: Compressing objects:   6% (57/944)
2024-06-05T08:45:23.5137987Z remote: Compressing objects:   7% (67/944)
2024-06-05T08:45:23.5161731Z remote: Compressing objects:   8% (76/944)
2024-06-05T08:45:23.5179114Z remote: Compressing objects:   9% (85/944)
2024-06-05T08:45:23.5189577Z remote: Compressing objects:  10% (95/944)
2024-06-05T08:45:23.5295394Z remote: Compressing objects:  11% (104/944)
2024-06-05T08:45:23.5368314Z remote: Compressing objects:  12% (114/944)
2024-06-05T08:45:23.5462223Z remote: Compressing objects:  13% (123/944)
2024-06-05T08:45:23.5548679Z remote: Compressing objects:  14% (133/944)
2024-06-05T08:45:23.5629687Z remote: Compressing objects:  15% (142/944)
2024-06-05T08:45:23.5695442Z remote: Compressing objects:  16% (152/944)
2024-06-05T08:45:23.5749108Z remote: Compressing objects:  17% (161/944)
2024-06-05T08:45:23.5819149Z remote: Compressing objects:  18% (170/944)
2024-06-05T08:45:23.5870021Z remote: Compressing objects:  19% (180/944)
2024-06-05T08:45:23.5935290Z remote: Compressing objects:  20% (189/944)
2024-06-05T08:45:23.5972592Z remote: Compressing objects:  21% (199/944)
2024-06-05T08:45:23.5999441Z remote: Compressing objects:  22% (208/944)
2024-06-05T08:45:23.6044046Z remote: Compressing objects:  23% (218/944)
2024-06-05T08:45:23.6073400Z remote: Compressing objects:  24% (227/944)
2024-06-05T08:45:23.6099543Z remote: Compressing objects:  25% (236/944)
2024-06-05T08:45:23.6126237Z remote: Compressing objects:  26% (246/944)
2024-06-05T08:45:23.6143416Z remote: Compressing objects:  27% (255/944)
2024-06-05T08:45:23.6173378Z remote: Compressing objects:  28% (265/944)
2024-06-05T08:45:23.6190335Z remote: Compressing objects:  29% (274/944)
2024-06-05T08:45:23.6205176Z remote: Compressing objects:  30% (284/944)
2024-06-05T08:45:23.6217275Z remote: Compressing objects:  31% (293/944)
2024-06-05T08:45:23.6224209Z remote: Compressing objects:  32% (303/944)
2024-06-05T08:45:23.6233941Z remote: Compressing objects:  33% (312/944)
2024-06-05T08:45:23.6236310Z remote: Compressing objects:  34% (321/944)
2024-06-05T08:45:23.6238897Z remote: Compressing objects:  35% (331/944)
2024-06-05T08:45:23.6241403Z remote: Compressing objects:  36% (340/944)
2024-06-05T08:45:23.6242093Z remote: Compressing objects:  37% (350/944)
2024-06-05T08:45:23.6242726Z remote: Compressing objects:  38% (359/944)
2024-06-05T08:45:23.6243310Z remote: Compressing objects:  39% (369/944)
2024-06-05T08:45:23.6244115Z remote: Compressing objects:  40% (378/944)
2024-06-05T08:45:23.6246921Z remote: Compressing objects:  41% (388/944)
2024-06-05T08:45:23.6249436Z remote: Compressing objects:  42% (397/944)
2024-06-05T08:45:23.6252002Z remote: Compressing objects:  43% (406/944)
2024-06-05T08:45:23.6254507Z remote: Compressing objects:  44% (416/944)
2024-06-05T08:45:23.6260104Z remote: Compressing objects:  45% (425/944)
2024-06-05T08:45:23.6265696Z remote: Compressing objects:  46% (435/944)
2024-06-05T08:45:23.6269011Z remote: Compressing objects:  47% (444/944)
2024-06-05T08:45:23.6273065Z remote: Compressing objects:  48% (454/944)
2024-06-05T08:45:23.6275747Z remote: Compressing objects:  49% (463/944)
2024-06-05T08:45:23.6279146Z remote: Compressing objects:  50% (472/944)
2024-06-05T08:45:23.6283110Z remote: Compressing objects:  51% (482/944)
2024-06-05T08:45:23.6287043Z remote: Compressing objects:  52% (491/944)
2024-06-05T08:45:23.6292132Z remote: Compressing objects:  53% (501/944)
2024-06-05T08:45:23.6295129Z remote: Compressing objects:  54% (510/944)
2024-06-05T08:45:23.6301850Z remote: Compressing objects:  55% (520/944)
2024-06-05T08:45:23.6304261Z remote: Compressing objects:  56% (529/944)
2024-06-05T08:45:23.6304877Z remote: Compressing objects:  57% (539/944)
2024-06-05T08:45:23.6307066Z remote: Compressing objects:  58% (548/944)
2024-06-05T08:45:23.6310535Z remote: Compressing objects:  59% (557/944)
2024-06-05T08:45:23.6314698Z remote: Compressing objects:  60% (567/944)
2024-06-05T08:45:23.6318527Z remote: Compressing objects:  61% (576/944)
2024-06-05T08:45:23.6320978Z remote: Compressing objects:  62% (586/944)
2024-06-05T08:45:23.6324356Z remote: Compressing objects:  63% (595/944)
2024-06-05T08:45:23.6326839Z remote: Compressing objects:  64% (605/944)
2024-06-05T08:45:23.6329525Z remote: Compressing objects:  65% (614/944)
2024-06-05T08:45:23.6334401Z remote: Compressing objects:  66% (624/944)
2024-06-05T08:45:23.6334981Z remote: Compressing objects:  67% (633/944)
2024-06-05T08:45:23.6352652Z remote: Compressing objects:  68% (642/944)
2024-06-05T08:45:23.6353263Z remote: Compressing objects:  69% (652/944)
2024-06-05T08:45:23.6353839Z remote: Compressing objects:  70% (661/944)
2024-06-05T08:45:23.6365399Z remote: Compressing objects:  71% (671/944)
2024-06-05T08:45:23.6365974Z remote: Compressing objects:  72% (680/944)
2024-06-05T08:45:23.6366791Z remote: Compressing objects:  73% (690/944)
2024-06-05T08:45:23.6367575Z remote: Compressing objects:  74% (699/944)
2024-06-05T08:45:23.6368155Z remote: Compressing objects:  75% (708/944)
2024-06-05T08:45:23.6368727Z remote: Compressing objects:  76% (718/944)
2024-06-05T08:45:23.6369295Z remote: Compressing objects:  77% (727/944)
2024-06-05T08:45:23.6369863Z remote: Compressing objects:  78% (737/944)
2024-06-05T08:45:23.6370426Z remote: Compressing objects:  79% (746/944)
2024-06-05T08:45:23.6370983Z remote: Compressing objects:  80% (756/944)
2024-06-05T08:45:23.6371552Z remote: Compressing objects:  81% (765/944)
2024-06-05T08:45:23.6372125Z remote: Compressing objects:  82% (775/944)
2024-06-05T08:45:23.6372834Z remote: Compressing objects:  83% (784/944)
2024-06-05T08:45:23.6373403Z remote: Compressing objects:  84% (793/944)
2024-06-05T08:45:23.6374075Z remote: Compressing objects:  85% (803/944)
2024-06-05T08:45:23.6374854Z remote: Compressing objects:  86% (812/944)
2024-06-05T08:45:23.6375439Z remote: Compressing objects:  87% (822/944)
2024-06-05T08:45:23.6375997Z remote: Compressing objects:  88% (831/944)
2024-06-05T08:45:23.6376685Z remote: Compressing objects:  89% (841/944)
2024-06-05T08:45:23.6377283Z remote: Compressing objects:  90% (850/944)
2024-06-05T08:45:23.6378060Z remote: Compressing objects:  91% (860/944)
2024-06-05T08:45:23.6378712Z remote: Compressing objects:  92% (869/944)
2024-06-05T08:45:23.6379280Z remote: Compressing objects:  93% (878/944)
2024-06-05T08:45:23.6379845Z remote: Compressing objects:  94% (888/944)
2024-06-05T08:45:23.6380414Z remote: Compressing objects:  95% (897/944)
2024-06-05T08:45:23.6380975Z remote: Compressing objects:  96% (907/944)
2024-06-05T08:45:23.6381687Z remote: Compressing objects:  97% (916/944)
2024-06-05T08:45:23.6382254Z remote: Compressing objects:  98% (926/944)
2024-06-05T08:45:23.6382823Z remote: Compressing objects:  99% (935/944)
2024-06-05T08:45:23.6383389Z remote: Compressing objects: 100% (944/944)
2024-06-05T08:45:23.6383995Z remote: Compressing objects: 100% (944/944), done.
2024-06-05T08:45:23.6454738Z Receiving objects:   0% (1/29467)
2024-06-05T08:45:23.6502384Z Receiving objects:   1% (295/29467)
2024-06-05T08:45:23.6612681Z Receiving objects:   2% (590/29467)
2024-06-05T08:45:23.6637893Z Receiving objects:   3% (885/29467)
2024-06-05T08:45:23.6740009Z Receiving objects:   4% (1179/29467)
2024-06-05T08:45:23.6782025Z Receiving objects:   5% (1474/29467)
2024-06-05T08:45:23.6870154Z Receiving objects:   6% (1769/29467)
2024-06-05T08:45:23.6895113Z Receiving objects:   7% (2063/29467)
2024-06-05T08:45:23.6925722Z Receiving objects:   8% (2358/29467)
2024-06-05T08:45:23.6961230Z Receiving objects:   9% (2653/29467)
2024-06-05T08:45:23.6992747Z Receiving objects:  10% (2947/29467)
2024-06-05T08:45:23.7022982Z Receiving objects:  11% (3242/29467)
2024-06-05T08:45:23.7054165Z Receiving objects:  12% (3537/29467)
2024-06-05T08:45:23.7092414Z Receiving objects:  13% (3831/29467)
2024-06-05T08:45:23.7123664Z Receiving objects:  14% (4126/29467)
2024-06-05T08:45:23.7154156Z Receiving objects:  15% (4421/29467)
2024-06-05T08:45:23.7192490Z Receiving objects:  16% (4715/29467)
2024-06-05T08:45:23.7232761Z Receiving objects:  17% (5010/29467)
2024-06-05T08:45:23.7275365Z Receiving objects:  18% (5305/29467)
2024-06-05T08:45:23.7397014Z Receiving objects:  19% (5599/29467)
2024-06-05T08:45:23.7527666Z Receiving objects:  20% (5894/29467)
2024-06-05T08:45:23.7568126Z Receiving objects:  21% (6189/29467)
2024-06-05T08:45:23.7593152Z Receiving objects:  22% (6483/29467)
2024-06-05T08:45:23.7609620Z Receiving objects:  23% (6778/29467)
2024-06-05T08:45:23.7632689Z Receiving objects:  24% (7073/29467)
2024-06-05T08:45:23.7662334Z Receiving objects:  25% (7367/29467)
2024-06-05T08:45:23.7674811Z Receiving objects:  26% (7662/29467)
2024-06-05T08:45:23.7693772Z Receiving objects:  27% (7957/29467)
2024-06-05T08:45:23.8218888Z Receiving objects:  28% (8251/29467)
2024-06-05T08:45:23.8339141Z Receiving objects:  29% (8546/29467)
2024-06-05T08:45:23.8368913Z Receiving objects:  30% (8841/29467)
2024-06-05T08:45:24.0243396Z Receiving objects:  31% (9135/29467)
2024-06-05T08:45:24.6395112Z Receiving objects:  32% (9430/29467)
2024-06-05T08:45:24.8780928Z Receiving objects:  32% (9565/29467), 74.38 MiB | 74.38 MiB/s
2024-06-05T08:45:25.3555565Z Receiving objects:  33% (9725/29467), 74.38 MiB | 74.38 MiB/s
2024-06-05T08:45:25.4225667Z Receiving objects:  34% (10019/29467), 113.44 MiB | 75.63 MiB/s
2024-06-05T08:45:25.4901386Z Receiving objects:  35% (10314/29467), 113.44 MiB | 75.63 MiB/s
2024-06-05T08:45:25.5569888Z Receiving objects:  36% (10609/29467), 113.44 MiB | 75.63 MiB/s
2024-06-05T08:45:25.6246736Z Receiving objects:  37% (10903/29467), 113.44 MiB | 75.63 MiB/s
2024-06-05T08:45:25.6395197Z Receiving objects:  38% (11198/29467), 113.44 MiB | 75.63 MiB/s
2024-06-05T08:45:25.6918227Z Receiving objects:  38% (11262/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:25.7587093Z Receiving objects:  39% (11493/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:25.8257963Z Receiving objects:  40% (11787/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:26.0634870Z Receiving objects:  41% (12082/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:26.1373574Z Receiving objects:  42% (12377/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:26.1492255Z Receiving objects:  43% (12671/29467), 152.04 MiB | 76.02 MiB/s
2024-06-05T08:45:26.1796134Z Receiving objects:  44% (12966/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.3709794Z Receiving objects:  45% (13261/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6043132Z Receiving objects:  46% (13555/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6239070Z Receiving objects:  47% (13850/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6249282Z Receiving objects:  48% (14145/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6263546Z Receiving objects:  49% (14439/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6272265Z Receiving objects:  50% (14734/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6281285Z Receiving objects:  51% (15029/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6396464Z Receiving objects:  52% (15323/29467), 186.65 MiB | 74.69 MiB/s
2024-06-05T08:45:26.6413713Z Receiving objects:  52% (15361/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6419109Z Receiving objects:  53% (15618/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6424053Z Receiving objects:  54% (15913/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6431418Z Receiving objects:  55% (16207/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6437662Z Receiving objects:  56% (16502/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6451752Z Receiving objects:  57% (16797/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6847424Z Receiving objects:  58% (17091/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6854335Z Receiving objects:  59% (17386/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6863934Z Receiving objects:  60% (17681/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6876306Z Receiving objects:  61% (17975/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6890234Z Receiving objects:  62% (18270/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6899330Z Receiving objects:  63% (18565/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6906719Z Receiving objects:  64% (18859/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6925291Z Receiving objects:  65% (19154/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6932116Z Receiving objects:  66% (19449/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.6939533Z Receiving objects:  67% (19743/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.8952633Z Receiving objects:  68% (20038/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9162150Z Receiving objects:  69% (20333/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9171527Z Receiving objects:  70% (20627/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9180177Z Receiving objects:  71% (20922/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9193494Z Receiving objects:  72% (21217/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9199796Z Receiving objects:  73% (21511/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9361154Z Receiving objects:  74% (21806/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9367293Z Receiving objects:  75% (22101/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9375705Z Receiving objects:  76% (22395/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9382185Z Receiving objects:  77% (22690/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9388453Z Receiving objects:  78% (22985/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9394468Z Receiving objects:  79% (23279/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9515404Z Receiving objects:  80% (23574/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9531622Z Receiving objects:  81% (23869/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9541669Z Receiving objects:  82% (24163/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9562593Z Receiving objects:  83% (24458/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9582548Z Receiving objects:  84% (24753/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9587294Z Receiving objects:  85% (25047/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9590922Z Receiving objects:  86% (25342/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9596283Z Receiving objects:  87% (25637/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9601641Z Receiving objects:  88% (25931/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9606618Z Receiving objects:  89% (26226/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9608519Z Receiving objects:  90% (26521/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9612390Z Receiving objects:  91% (26815/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9617751Z Receiving objects:  92% (27110/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9651987Z Receiving objects:  93% (27405/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9729043Z Receiving objects:  94% (27699/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9765953Z Receiving objects:  95% (27994/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9799660Z Receiving objects:  96% (28289/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9817797Z Receiving objects:  97% (28583/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9836373Z Receiving objects:  98% (28878/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9866323Z Receiving objects:  99% (29173/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9868353Z remote: Total 29467 (delta 4123), reused 4064 (delta 3751), pack-reused 24770
2024-06-05T08:45:26.9883425Z Receiving objects: 100% (29467/29467), 225.81 MiB | 75.29 MiB/s
2024-06-05T08:45:26.9884136Z Receiving objects: 100% (29467/29467), 244.97 MiB | 73.16 MiB/s, done.
2024-06-05T08:45:26.9912870Z Resolving deltas:   0% (0/15963)
2024-06-05T08:45:26.9939736Z Resolving deltas:   1% (178/15963)
2024-06-05T08:45:26.9971512Z Resolving deltas:   2% (335/15963)
2024-06-05T08:45:27.0014575Z Resolving deltas:   3% (486/15963)
2024-06-05T08:45:27.0047797Z Resolving deltas:   4% (640/15963)
2024-06-05T08:45:27.0062712Z Resolving deltas:   5% (801/15963)
2024-06-05T08:45:27.0075667Z Resolving deltas:   6% (969/15963)
2024-06-05T08:45:27.0083826Z Resolving deltas:   7% (1124/15963)
2024-06-05T08:45:27.0199531Z Resolving deltas:   8% (1279/15963)
2024-06-05T08:45:27.0219759Z Resolving deltas:   9% (1446/15963)
2024-06-05T08:45:27.0258118Z Resolving deltas:  10% (1604/15963)
2024-06-05T08:45:27.0263184Z Resolving deltas:  11% (1896/15963)
2024-06-05T08:45:27.0270593Z Resolving deltas:  12% (1965/15963)
2024-06-05T08:45:27.0280568Z Resolving deltas:  13% (2078/15963)
2024-06-05T08:45:27.0303262Z Resolving deltas:  14% (2238/15963)
2024-06-05T08:45:27.0321608Z Resolving deltas:  15% (2395/15963)
2024-06-05T08:45:27.0338628Z Resolving deltas:  16% (2555/15963)
2024-06-05T08:45:27.0353247Z Resolving deltas:  17% (2720/15963)
2024-06-05T08:45:27.0362857Z Resolving deltas:  18% (2874/15963)
2024-06-05T08:45:27.0373769Z Resolving deltas:  19% (3033/15963)
2024-06-05T08:45:27.0383359Z Resolving deltas:  20% (3200/15963)
2024-06-05T08:45:27.0391213Z Resolving deltas:  21% (3363/15963)
2024-06-05T08:45:27.0399401Z Resolving deltas:  22% (3513/15963)
2024-06-05T08:45:27.0410197Z Resolving deltas:  23% (3673/15963)
2024-06-05T08:45:27.0419123Z Resolving deltas:  24% (3832/15963)
2024-06-05T08:45:27.0438600Z Resolving deltas:  25% (3994/15963)
2024-06-05T08:45:27.0447984Z Resolving deltas:  26% (4151/15963)
2024-06-05T08:45:27.0454005Z Resolving deltas:  27% (4311/15963)
2024-06-05T08:45:27.0460873Z Resolving deltas:  28% (4471/15963)
2024-06-05T08:45:27.0466789Z Resolving deltas:  29% (4634/15963)
2024-06-05T08:45:27.0472736Z Resolving deltas:  30% (4790/15963)
2024-06-05T08:45:27.0488384Z Resolving deltas:  31% (4952/15963)
2024-06-05T08:45:27.0497366Z Resolving deltas:  32% (5111/15963)
2024-06-05T08:45:27.0507957Z Resolving deltas:  33% (5270/15963)
2024-06-05T08:45:27.0516482Z Resolving deltas:  34% (5439/15963)
2024-06-05T08:45:27.0525946Z Resolving deltas:  35% (5603/15963)
2024-06-05T08:45:27.0534058Z Resolving deltas:  36% (5751/15963)
2024-06-05T08:45:27.0544058Z Resolving deltas:  37% (5962/15963)
2024-06-05T08:45:27.0545949Z Resolving deltas:  38% (6194/15963)
2024-06-05T08:45:27.0554987Z Resolving deltas:  39% (6236/15963)
2024-06-05T08:45:27.0559752Z Resolving deltas:  40% (6490/15963)
2024-06-05T08:45:27.0572941Z Resolving deltas:  41% (6562/15963)
2024-06-05T08:45:27.0578735Z Resolving deltas:  42% (6752/15963)
2024-06-05T08:45:27.0583819Z Resolving deltas:  43% (6866/15963)
2024-06-05T08:45:27.0591436Z Resolving deltas:  44% (7025/15963)
2024-06-05T08:45:27.0598968Z Resolving deltas:  45% (7193/15963)
2024-06-05T08:45:27.0614980Z Resolving deltas:  46% (7343/15963)
2024-06-05T08:45:27.0623322Z Resolving deltas:  47% (7505/15963)
2024-06-05T08:45:27.0632194Z Resolving deltas:  48% (7679/15963)
2024-06-05T08:45:27.0649962Z Resolving deltas:  49% (7822/15963)
2024-06-05T08:45:27.0660134Z Resolving deltas:  50% (7982/15963)
2024-06-05T08:45:27.0670152Z Resolving deltas:  51% (8150/15963)
2024-06-05T08:45:27.0676638Z Resolving deltas:  52% (8312/15963)
2024-06-05T08:45:27.0684815Z Resolving deltas:  53% (8466/15963)
2024-06-05T08:45:27.0689735Z Resolving deltas:  54% (8624/15963)
2024-06-05T08:45:27.0695324Z Resolving deltas:  55% (8791/15963)
2024-06-05T08:45:27.0701817Z Resolving deltas:  56% (8940/15963)
2024-06-05T08:45:27.0710650Z Resolving deltas:  57% (9103/15963)
2024-06-05T08:45:27.0717591Z Resolving deltas:  58% (9273/15963)
2024-06-05T08:45:27.0723679Z Resolving deltas:  59% (9449/15963)
2024-06-05T08:45:27.0735995Z Resolving deltas:  60% (9599/15963)
2024-06-05T08:45:27.0741089Z Resolving deltas:  61% (9807/15963)
2024-06-05T08:45:27.0749375Z Resolving deltas:  62% (9917/15963)
2024-06-05T08:45:27.0755808Z Resolving deltas:  63% (10059/15963)
2024-06-05T08:45:27.0765267Z Resolving deltas:  64% (10224/15963)
2024-06-05T08:45:27.0774705Z Resolving deltas:  65% (10389/15963)
2024-06-05T08:45:27.0785910Z Resolving deltas:  66% (10537/15963)
2024-06-05T08:45:27.0796343Z Resolving deltas:  67% (10701/15963)
2024-06-05T08:45:27.0801006Z Resolving deltas:  68% (10907/15963)
2024-06-05T08:45:27.0808142Z Resolving deltas:  69% (11023/15963)
2024-06-05T08:45:27.0813533Z Resolving deltas:  70% (11189/15963)
2024-06-05T08:45:27.0821151Z Resolving deltas:  71% (11335/15963)
2024-06-05T08:45:27.0825390Z Resolving deltas:  72% (11532/15963)
2024-06-05T08:45:27.0831650Z Resolving deltas:  73% (11658/15963)
2024-06-05T08:45:27.0842895Z Resolving deltas:  74% (11818/15963)
2024-06-05T08:45:27.0857828Z Resolving deltas:  75% (11987/15963)
2024-06-05T08:45:27.0865854Z Resolving deltas:  76% (12211/15963)
2024-06-05T08:45:27.0870183Z Resolving deltas:  77% (12386/15963)
2024-06-05T08:45:27.0879063Z Resolving deltas:  78% (12474/15963)
2024-06-05T08:45:27.0885287Z Resolving deltas:  79% (12660/15963)
2024-06-05T08:45:27.0907500Z Resolving deltas:  80% (12816/15963)
2024-06-05T08:45:27.0915971Z Resolving deltas:  83% (13284/15963)
2024-06-05T08:45:27.0920044Z Resolving deltas:  84% (13478/15963)
2024-06-05T08:45:27.0928558Z Resolving deltas:  85% (13569/15963)
2024-06-05T08:45:27.0938054Z Resolving deltas:  86% (13751/15963)
2024-06-05T08:45:27.0948967Z Resolving deltas:  87% (13897/15963)
2024-06-05T08:45:27.0954246Z Resolving deltas:  88% (14143/15963)
2024-06-05T08:45:27.0956881Z Resolving deltas:  89% (14276/15963)
2024-06-05T08:45:27.0973877Z Resolving deltas:  90% (14368/15963)
2024-06-05T08:45:27.0995415Z Resolving deltas:  91% (14530/15963)
2024-06-05T08:45:27.1008228Z Resolving deltas:  92% (14698/15963)
2024-06-05T08:45:27.1021929Z Resolving deltas:  93% (14866/15963)
2024-06-05T08:45:27.1040467Z Resolving deltas:  94% (15036/15963)
2024-06-05T08:45:27.1090246Z Resolving deltas:  95% (15223/15963)
2024-06-05T08:45:27.1099000Z Resolving deltas:  96% (15422/15963)
2024-06-05T08:45:27.1117570Z Resolving deltas:  97% (15497/15963)
2024-06-05T08:45:27.1131226Z Resolving deltas:  98% (15667/15963)
2024-06-05T08:45:27.1156080Z Resolving deltas:  99% (15825/15963)
2024-06-05T08:45:27.1156618Z Resolving deltas: 100% (15963/15963)
2024-06-05T08:45:27.1157104Z Resolving deltas: 100% (15963/15963), done.
2024-06-05T08:45:28.1721078Z + pushd torchbench
2024-06-05T08:45:28.1721594Z ~/workspace/torchbench ~/workspace
2024-06-05T08:45:28.1722239Z + git checkout d6015d42d9a1834bc7595c4bd6852562fb80b30b
2024-06-05T08:45:28.2239114Z Note: switching to 'd6015d42d9a1834bc7595c4bd6852562fb80b30b'.
2024-06-05T08:45:28.2239635Z 
2024-06-05T08:45:28.2240207Z You are in 'detached HEAD' state. You can look around, make experimental
2024-06-05T08:45:28.2241320Z changes and commit them, and you can discard any commits you make in this
2024-06-05T08:45:28.2242891Z state without impacting any branches by switching back to a branch.
2024-06-05T08:45:28.2243473Z 
2024-06-05T08:45:28.2243811Z If you want to create a new branch to retain commits you create, you may
2024-06-05T08:45:28.2244678Z do so (now or later) by using -c with the switch command. Example:
2024-06-05T08:45:28.2245131Z 
2024-06-05T08:45:28.2245334Z   git switch -c <new-branch-name>
2024-06-05T08:45:28.2245630Z 
2024-06-05T08:45:28.2245784Z Or undo this operation with:
2024-06-05T08:45:28.2246060Z 
2024-06-05T08:45:28.2246189Z   git switch -
2024-06-05T08:45:28.2246386Z 
2024-06-05T08:45:28.2247040Z Turn off this advice by setting config variable advice.detachedHead to false
2024-06-05T08:45:28.2247565Z 
2024-06-05T08:45:28.2247814Z HEAD is now at d6015d42 Add sam_fast torchbench (#2182)
2024-06-05T08:45:28.2248362Z + '[' '' ']'
2024-06-05T08:45:28.2248887Z + python install.py --continue_on_fail
2024-06-05T08:45:30.1517913Z checking packages torch, torchvision, torchaudio are installed...OK
2024-06-05T08:45:49.5184197Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2024-06-05T08:45:51.9557807Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2024-06-05T08:46:11.8979555Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch...OK
2024-06-05T08:46:14.6869969Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2024-06-05T08:46:17.2117953Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2024-06-05T08:46:17.2270322Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2024-06-05T08:46:26.1949908Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2024-06-05T08:46:29.7001771Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2024-06-05T08:46:33.2446136Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2024-06-05T08:46:36.7393990Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2024-06-05T08:46:36.7395652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2024-06-05T08:46:38.7620477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2024-06-05T08:46:42.4757357Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2024-06-05T08:46:42.4911336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2024-06-05T08:47:35.1223880Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2024-06-05T08:47:44.8715977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2024-06-05T08:47:52.5454724Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2024-06-05T08:48:00.0142869Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2024-06-05T08:48:09.2171293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2024-06-05T08:48:16.9835772Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2024-06-05T08:48:24.1073745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2024-06-05T08:48:31.7540307Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2024-06-05T08:48:39.8596059Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2024-06-05T08:48:47.7377380Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2024-06-05T08:48:55.2817238Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2024-06-05T08:49:02.8472584Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2024-06-05T08:49:06.9735792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2024-06-05T08:49:50.6333525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2024-06-05T08:50:13.7358686Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2024-06-05T08:50:22.5050508Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2024-06-05T08:50:32.2142033Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2024-06-05T08:50:34.6697873Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2024-06-05T08:50:37.3645409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2024-06-05T08:50:44.1387436Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2024-06-05T08:50:52.6161315Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2024-06-05T08:50:59.7022071Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2024-06-05T08:51:08.7116956Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2024-06-05T08:51:16.1228962Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2024-06-05T08:51:22.7569468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2024-06-05T08:51:30.4812335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2024-06-05T08:51:47.1256464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2024-06-05T08:51:54.8279200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2024-06-05T08:52:00.5612345Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2024-06-05T08:52:07.0402801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2024-06-05T08:52:15.3378702Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2024-06-05T08:52:15.3380455Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2024-06-05T08:52:29.0744754Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2024-06-05T08:52:36.7391176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2024-06-05T08:52:36.7392986Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found
2024-06-05T08:52:45.7429904Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK
2024-06-05T08:52:48.2652319Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2024-06-05T08:52:50.8016836Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2024-06-05T08:53:36.0168231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2024-06-05T08:54:56.0478535Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK
2024-06-05T08:54:56.0638698Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2024-06-05T08:54:58.8329665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2024-06-05T08:54:58.8488941Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2024-06-05T08:54:58.8650760Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2024-06-05T08:54:58.8805755Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2024-06-05T08:54:58.8962420Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2024-06-05T08:54:58.9117795Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2024-06-05T08:55:19.6975186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK
2024-06-05T08:55:19.6976943Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found
2024-06-05T08:55:22.2411336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2024-06-05T08:55:25.8949484Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2024-06-05T08:55:25.9111780Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2024-06-05T08:55:25.9275993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2024-06-05T08:55:25.9430282Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2024-06-05T08:55:25.9588286Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2024-06-05T08:55:25.9748605Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2024-06-05T08:55:33.3063897Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2024-06-05T08:55:36.0298401Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2024-06-05T08:55:42.8421525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2024-06-05T08:55:42.8584889Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2024-06-05T08:55:42.8750362Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2024-06-05T08:55:42.8905313Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2024-06-05T08:55:42.9063590Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2024-06-05T08:55:42.9222503Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2024-06-05T08:55:53.2956335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2024-06-05T08:56:05.3113372Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK
2024-06-05T08:56:05.3280300Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2024-06-05T08:56:05.3283852Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found
2024-06-05T08:56:05.3287618Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found
2024-06-05T08:56:08.9447552Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2024-06-05T08:56:11.8666714Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2024-06-05T08:56:11.8830822Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2024-06-05T08:56:36.0792896Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK
2024-06-05T08:56:43.3913563Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK
2024-06-05T08:56:51.1100117Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2024-06-05T08:57:10.7940386Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2024-06-05T08:57:10.8102648Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2024-06-05T08:57:10.8269049Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2024-06-05T08:57:10.8433765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2024-06-05T08:57:10.8591259Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2024-06-05T08:57:10.8746774Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2024-06-05T08:57:10.8905546Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2024-06-05T08:57:10.9064877Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2024-06-05T08:57:25.8624541Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK
2024-06-05T08:57:41.2561372Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2024-06-05T08:57:41.2720707Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2024-06-05T08:57:46.6927962Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2024-06-05T08:57:50.1154123Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2024-06-05T08:57:50.5240222Z + popd
2024-06-05T08:57:50.5240688Z ~/workspace
2024-06-05T08:57:50.5241137Z + [[ inductor_torchbench != *cpu_inductor* ]]
2024-06-05T08:57:50.5242161Z + install_torchrec_and_fbgemm
2024-06-05T08:57:50.5243429Z + local torchrec_commit
2024-06-05T08:57:50.5243843Z ++ get_pinned_commit torchrec
2024-06-05T08:57:50.5244289Z ++ cat .github/ci_commit_pins/torchrec.txt
2024-06-05T08:57:50.5253278Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T08:57:50.5253839Z + local fbgemm_commit
2024-06-05T08:57:50.5255275Z ++ get_pinned_commit fbgemm
2024-06-05T08:57:50.5255724Z ++ cat .github/ci_commit_pins/fbgemm.txt
2024-06-05T08:57:50.5263731Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd
2024-06-05T08:57:50.5264437Z + pip_uninstall torchrec-nightly
2024-06-05T08:57:50.5264932Z + pip uninstall -y torchrec-nightly
2024-06-05T08:57:50.8830133Z WARNING: Skipping torchrec-nightly as it is not installed.
2024-06-05T08:57:50.9259522Z + pip_uninstall fbgemm-gpu-nightly
2024-06-05T08:57:50.9260128Z + pip uninstall -y fbgemm-gpu-nightly
2024-06-05T08:57:51.2783530Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2024-06-05T08:57:51.3243978Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2024-06-05T08:57:51.3245009Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2024-06-05T08:57:51.7219959Z Collecting setuptools-git-versioning
2024-06-05T08:57:51.7366116Z   Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB)
2024-06-05T08:57:51.7598623Z Collecting scikit-build
2024-06-05T08:57:51.7620881Z   Downloading scikit_build-0.17.6-py3-none-any.whl.metadata (14 kB)
2024-06-05T08:57:51.7818634Z Collecting pyre-extensions
2024-06-05T08:57:51.7842299Z   Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB)
2024-06-05T08:57:51.7915327Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.0)
2024-06-05T08:57:51.7918928Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1)
2024-06-05T08:57:51.8052712Z Collecting toml>=0.10.2 (from setuptools-git-versioning)
2024-06-05T08:57:51.8076743Z   Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
2024-06-05T08:57:51.8278640Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0)
2024-06-05T08:57:51.8285686Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1)
2024-06-05T08:57:51.8290739Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0)
2024-06-05T08:57:51.8440882Z Collecting typing-inspect (from pyre-extensions)
2024-06-05T08:57:51.8466288Z   Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
2024-06-05T08:57:51.8502383Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.1)
2024-06-05T08:57:51.8729863Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2024-06-05T08:57:51.8770724Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB)
2024-06-05T08:57:51.8835071Z Downloading scikit_build-0.17.6-py3-none-any.whl (84 kB)
2024-06-05T08:57:51.8909924Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB)
2024-06-05T08:57:51.8973904Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
2024-06-05T08:57:51.9040416Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
2024-06-05T08:57:54.2776191Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions
2024-06-05T08:57:54.3910000Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.17.6 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0
2024-06-05T08:57:54.5153047Z + CUDA_PATH=/usr/local/cuda-12.1
2024-06-05T08:57:54.5154351Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-06-05T08:57:54.5156288Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-06-05T08:57:54.8590821Z Collecting fbgemm-gpu
2024-06-05T08:57:54.8592967Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-kr842x2t/fbgemm-gpu_dbd5ffd030284867beb25e75d80ba004
2024-06-05T08:57:54.8610176Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-kr842x2t/fbgemm-gpu_dbd5ffd030284867beb25e75d80ba004
2024-06-05T08:57:55.7277500Z   Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd'
2024-06-05T08:57:55.7295934Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd
2024-06-05T08:57:56.1349813Z   Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd
2024-06-05T08:57:56.5889108Z   Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd
2024-06-05T08:57:56.5890712Z   Running command git submodule update --init --recursive -q
2024-06-05T08:58:03.8417831Z   Preparing metadata (setup.py) ... [?25l- done
2024-06-05T08:58:03.8440677Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.26.4)
2024-06-05T08:58:03.8460717Z Building wheels for collected packages: fbgemm-gpu
2024-06-05T09:39:31.0339475Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
2024-06-05T09:39:31.2145721Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253212727 sha256=a89164f86681e7c043a065aded1be656d7dc93ee2ce63024baa290f66c5d93a4
2024-06-05T09:39:31.2148222Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39
2024-06-05T09:39:31.2168075Z Successfully built fbgemm-gpu
2024-06-05T09:39:33.5846072Z Installing collected packages: fbgemm-gpu
2024-06-05T09:39:36.7230238Z Successfully installed fbgemm-gpu-0.4.1rc0.post421
2024-06-05T09:39:37.3500180Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:37.3501826Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:37.6946092Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:37.6951939Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-e0x6cvzo
2024-06-05T09:39:37.6968958Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-e0x6cvzo
2024-06-05T09:39:38.3597611Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2024-06-05T09:39:38.3615928Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:38.6392874Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:39.0536326Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-06-05T09:39:39.2907259Z   Preparing metadata (setup.py) ... [?25l- done
2024-06-05T09:39:39.2958185Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2024-06-05T09:39:39.2961428Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.2.2)
2024-06-05T09:39:39.2967769Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2024-06-05T09:39:39.2970868Z Requirement already satisfied: torchmetrics in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (1.4.0.post0)
2024-06-05T09:39:39.2974968Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4)
2024-06-05T09:39:39.2978689Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421)
2024-06-05T09:39:39.2992545Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.26.4)
2024-06-05T09:39:39.3014374Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.8.2)
2024-06-05T09:39:39.3747482Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0)
2024-06-05T09:39:39.3753372Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1)
2024-06-05T09:39:39.3758366Z Requirement already satisfied: tzdata>=2022.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1)
2024-06-05T09:39:39.4551339Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.0)
2024-06-05T09:39:39.4554940Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.4.0a0+gitdffed71)
2024-06-05T09:39:39.4560537Z Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (0.11.2)
2024-06-05T09:39:39.4683964Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1)
2024-06-05T09:39:39.4686641Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.1)
2024-06-05T09:39:39.4725788Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0)
2024-06-05T09:39:39.4793902Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1)
2024-06-05T09:39:39.4797883Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1)
2024-06-05T09:39:39.4800929Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8)
2024-06-05T09:39:39.4804270Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4)
2024-06-05T09:39:39.4807649Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0)
2024-06-05T09:39:39.5343134Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5)
2024-06-05T09:39:39.5782482Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0)
2024-06-05T09:39:39.5871183Z Building wheels for collected packages: torchrec
2024-06-05T09:39:40.0333546Z   Building wheel for torchrec (setup.py) ... [?25l- \ | done
2024-06-05T09:39:40.0342944Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=232b24587988be4fb1da540743c6df155404c27a2bfcb72b203330f9a048ca1d
2024-06-05T09:39:40.0344613Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2024-06-05T09:39:40.0369478Z Successfully built torchrec
2024-06-05T09:39:42.1919482Z Installing collected packages: torchrec
2024-06-05T09:39:42.4023143Z Successfully installed torchrec-0.3.2
2024-06-05T09:39:42.5338363Z ++ pwd
2024-06-05T09:39:42.5340494Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2024-06-05T09:39:42.5341264Z + test_dynamo_benchmark torchbench 1
2024-06-05T09:39:42.5343678Z ++ pwd
2024-06-05T09:39:42.5345264Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-06-05T09:39:42.5346054Z + local suite=torchbench
2024-06-05T09:39:42.5346600Z + shift
2024-06-05T09:39:42.5347017Z + local shard_id=1
2024-06-05T09:39:42.5347466Z + shift
2024-06-05T09:39:42.5347881Z + [[ inductor_torchbench == *perf_compare* ]]
2024-06-05T09:39:42.5348590Z + [[ inductor_torchbench == *perf* ]]
2024-06-05T09:39:42.5350162Z + [[ inductor_torchbench == *cpu_inductor* ]]
2024-06-05T09:39:42.5350906Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-06-05T09:39:42.5352168Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16
2024-06-05T09:39:42.5352870Z ++ pwd
2024-06-05T09:39:42.5353377Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-06-05T09:39:42.5354082Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-06-05T09:39:42.5376783Z + local name=inference
2024-06-05T09:39:42.5377204Z + shift
2024-06-05T09:39:42.5377517Z + local suite=torchbench
2024-06-05T09:39:42.5377950Z + shift
2024-06-05T09:39:42.5378257Z + local shard_id=1
2024-06-05T09:39:42.5378582Z + shift
2024-06-05T09:39:42.5378885Z + partition_flags=()
2024-06-05T09:39:42.5379248Z + local partition_flags
2024-06-05T09:39:42.5379741Z + [[ -n 2 ]]
2024-06-05T09:39:42.5380115Z + [[ -n 1 ]]
2024-06-05T09:39:42.5380754Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-06-05T09:39:42.5381487Z + [[ inductor_torchbench == *perf_compare* ]]
2024-06-05T09:39:42.5381988Z + [[ inductor_torchbench == *perf* ]]
2024-06-05T09:39:42.5382489Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-06-05T09:39:42.5384188Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv
2024-06-05T09:39:47.9098869Z 
2024-06-05T09:39:48.3142016Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:39:48.3142588Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:39:48.3145628Z cuda eval  lennard_jones                      
2024-06-05T09:39:52.1533694Z E0605 09:39:52.152000 140630602224256 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00022, (ref-fp64): 0.00059 and shape=torch.Size([4, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:39:52.1535070Z pass
2024-06-05T09:39:52.1547888Z TIMING: code_gen:0.82928 inductor_compile:0.93248 backend_compile:2.52908 entire_frame_compile:2.69126
2024-06-05T09:39:52.1549363Z STATS: call_* op count: 18 | FakeTensor.__torch_dispatch__:119 | FakeTensorMode.__torch_dispatch__:1268 | ProxyTorchDispatchMode.__torch_dispatch__:339
2024-06-05T09:39:52.1550611Z Dynamo produced 1 graphs covering 18 ops with 0 graph breaks (0 unique)
2024-06-05T09:39:55.6986919Z 
2024-06-05T09:39:56.2126935Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:39:56.2128002Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:39:56.2161494Z cuda eval  llama                              
2024-06-05T09:40:14.1179870Z E0605 09:40:14.117000 140489301754496 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00264, (ref-fp64): 0.00461 and shape=torch.Size([4, 32000]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.000100
2024-06-05T09:40:14.1186617Z pass
2024-06-05T09:40:14.1334476Z TIMING: code_gen:4.8677 inductor_compile:8.42578 backend_compile:14.47297 entire_frame_compile:16.32244
2024-06-05T09:40:14.1337264Z STATS: call_* op count: 1058 | FakeTensor.__torch_dispatch__:2971 | FakeTensorMode.__torch_dispatch__:34594 | ProxyTorchDispatchMode.__torch_dispatch__:10235
2024-06-05T09:40:14.1338483Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique)
2024-06-05T09:40:18.0902809Z 
2024-06-05T09:40:18.3639819Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:18.3640489Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:18.3641058Z cuda eval  llama_v2_7b_16h                    
2024-06-05T09:40:18.3646133Z Traceback (most recent call last):
2024-06-05T09:40:18.3647294Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T09:40:18.3648094Z     ) = runner.load_model(
2024-06-05T09:40:18.3648901Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model
2024-06-05T09:40:18.3650729Z     benchmark = benchmark_cls(
2024-06-05T09:40:18.3651805Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T09:40:18.3660639Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T09:40:18.3661926Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__
2024-06-05T09:40:18.3663210Z     super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:40:18.3664575Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-06-05T09:40:18.3665825Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:40:18.3666868Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-06-05T09:40:18.3667684Z     self._skip_by_device_name()
2024-06-05T09:40:18.3671268Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-06-05T09:40:18.3672683Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-06-05T09:40:18.3673913Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`.
2024-06-05T09:40:18.3674577Z 
2024-06-05T09:40:18.3674726Z model_fail_to_load
2024-06-05T09:40:21.5826067Z 
2024-06-05T09:40:21.8580631Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:21.8581231Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:21.8581707Z cuda eval  llava                              
2024-06-05T09:40:21.8586656Z Traceback (most recent call last):
2024-06-05T09:40:21.8587671Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T09:40:21.8588605Z     ) = runner.load_model(
2024-06-05T09:40:21.8589425Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model
2024-06-05T09:40:21.8590370Z     benchmark = benchmark_cls(
2024-06-05T09:40:21.8591304Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T09:40:21.8592474Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T09:40:21.8593414Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__
2024-06-05T09:40:21.8594617Z     super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:40:21.8595959Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-06-05T09:40:21.8597237Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:40:21.8598297Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-06-05T09:40:21.8599121Z     self._skip_by_device_name()
2024-06-05T09:40:21.8599992Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-06-05T09:40:21.8601309Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-06-05T09:40:21.8602664Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2024-06-05T09:40:21.8603532Z 
2024-06-05T09:40:21.8603677Z model_fail_to_load
2024-06-05T09:40:25.0036441Z 
2024-06-05T09:40:25.6253955Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:25.6254634Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:25.6256686Z cuda eval  maml                               
2024-06-05T09:40:25.6259417Z pass_due_to_skip
2024-06-05T09:40:25.6267985Z TIMING:
2024-06-05T09:40:25.6268313Z STATS: call_* op count: 0
2024-06-05T09:40:25.6268983Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-06-05T09:40:28.8894324Z 
2024-06-05T09:40:29.3907025Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:29.3907537Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:29.3913865Z cuda eval  maml_omniglot                      
2024-06-05T09:40:33.3306367Z E0605 09:40:33.329000 140705398637184 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00035, (ref-fp64): 0.00030 and shape=torch.Size([5, 5]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:40:33.3307819Z pass
2024-06-05T09:40:33.3318699Z TIMING: code_gen:1.18929 inductor_compile:1.43185 backend_compile:2.58478 entire_frame_compile:2.75454
2024-06-05T09:40:33.3320305Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:271 | FakeTensorMode.__torch_dispatch__:2380 | ProxyTorchDispatchMode.__torch_dispatch__:555
2024-06-05T09:40:33.3321520Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique)
2024-06-05T09:40:36.8118389Z 
2024-06-05T09:40:37.0817165Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth
2024-06-05T09:40:37.0989356Z 
2024-06-05T09:40:37.0989912Z 
2024-06-05T09:40:37.1443870Z   0% 0.00/16.9M [00:00<?, ?B/s]
2024-06-05T09:40:37.1444364Z 100% 16.9M/16.9M [00:00<00:00, 391MB/s]
2024-06-05T09:40:37.7591532Z 
2024-06-05T09:40:37.7592372Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:37.7654516Z cuda eval  mnasnet1_0                         
2024-06-05T09:40:51.4588573Z E0605 09:40:51.457000 139754462798464 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01948, (ref-fp64): 0.01890 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:40:51.4590110Z pass
2024-06-05T09:40:51.4616067Z TIMING: code_gen:2.31917 inductor_compile:5.17976 backend_compile:9.61867 entire_frame_compile:11.11505
2024-06-05T09:40:51.4617490Z STATS: call_* op count: 304 | FakeTensor.__torch_dispatch__:3728 | FakeTensorMode.__torch_dispatch__:27505 | ProxyTorchDispatchMode.__torch_dispatch__:7025
2024-06-05T09:40:51.4618692Z Dynamo produced 1 graphs covering 304 ops with 0 graph breaks (0 unique)
2024-06-05T09:40:55.2639186Z 
2024-06-05T09:40:56.1744459Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:56.1745074Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:40:56.1807181Z cuda eval  mobilenet_v2                       
2024-06-05T09:41:09.5120183Z E0605 09:41:09.511000 140442450109056 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.02172, (ref-fp64): 0.02268 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:41:09.5121646Z pass
2024-06-05T09:41:09.5148576Z TIMING: code_gen:2.21378 inductor_compile:5.0104 backend_compile:9.64226 entire_frame_compile:11.30201
2024-06-05T09:41:09.5150072Z STATS: call_* op count: 306 | FakeTensor.__torch_dispatch__:3744 | FakeTensorMode.__torch_dispatch__:28414 | ProxyTorchDispatchMode.__torch_dispatch__:7953
2024-06-05T09:41:09.5151370Z Dynamo produced 1 graphs covering 306 ops with 0 graph breaks (0 unique)
2024-06-05T09:41:13.3682539Z 
2024-06-05T09:41:13.5749312Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:41:13.5750069Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:41:13.5750759Z cuda eval  mobilenet_v2_quantized_qat         
2024-06-05T09:41:13.5752674Z Traceback (most recent call last):
2024-06-05T09:41:13.5753652Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T09:41:13.5755655Z     ) = runner.load_model(
2024-06-05T09:41:13.5756644Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model
2024-06-05T09:41:13.5757721Z     benchmark = benchmark_cls(
2024-06-05T09:41:13.5758586Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T09:41:13.5759557Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T09:41:13.5760732Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat/__init__.py", line 21, in __init__
2024-06-05T09:41:13.5762146Z     raise NotImplementedError("The eval test only supports CPU.")
2024-06-05T09:41:13.5763092Z NotImplementedError: The eval test only supports CPU.
2024-06-05T09:41:13.5763512Z 
2024-06-05T09:41:13.5765973Z model_fail_to_load
2024-06-05T09:41:16.7213174Z 
2024-06-05T09:41:17.7754744Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:41:17.7755255Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:41:17.7817275Z cuda eval  mobilenet_v3_large                 
2024-06-05T09:41:33.7286881Z E0605 09:41:33.727000 139812265845376 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.02804, (ref-fp64): 0.02425 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:41:33.7288260Z pass
2024-06-05T09:41:33.7303430Z TIMING: code_gen:3.60931 inductor_compile:6.80437 backend_compile:11.6766 entire_frame_compile:13.51005
2024-06-05T09:41:33.7304925Z STATS: call_* op count: 374 | FakeTensor.__torch_dispatch__:4027 | FakeTensorMode.__torch_dispatch__:28696 | ProxyTorchDispatchMode.__torch_dispatch__:8087
2024-06-05T09:41:33.7306237Z Dynamo produced 1 graphs covering 374 ops with 0 graph breaks (0 unique)
2024-06-05T09:41:37.6786678Z 
2024-06-05T09:41:39.3551197Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:41:39.3551803Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:41:39.3669980Z cuda eval  moco                               
2024-06-05T09:41:42.0801349Z [rank0]:W0605 09:41:42.079000 139634400522880 torch/_logging/_internal.py:1033] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T09:41:49.3086857Z [rank0]:W0605 09:41:49.307000 139634400522880 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2024-06-05T09:41:49.3275360Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2024-06-05T09:41:49.3281276Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐
2024-06-05T09:41:49.3283043Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2024-06-05T09:41:49.3284682Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤
2024-06-05T09:41:49.3286249Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] │       0 │         157 │               44910720 │
2024-06-05T09:41:49.3288129Z [rank0]:W0605 09:41:49.326000 139634400522880 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘
2024-06-05T09:42:07.4115070Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 
2024-06-05T09:42:07.4116695Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward
2024-06-05T09:42:07.4117934Z     self._momentum_update_key_encoder()  # update the key encoder
2024-06-05T09:42:07.4119347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2024-06-05T09:42:07.4120489Z     return func(*args, **kwargs)
2024-06-05T09:42:07.4121504Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder
2024-06-05T09:42:07.4122809Z     param_k.mul_(self.m).add_(param_q.mul(1. - self.m))
2024-06-05T09:42:07.4123205Z 
2024-06-05T09:42:08.3172934Z [rank0]:W0605 09:42:08.316000 139634400522880 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T09:42:08.3274549Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 
2024-06-05T09:42:08.3276902Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp
2024-06-05T09:42:08.3277986Z     idx_shuffle = torch.randperm(batch_size_all).cuda()
2024-06-05T09:42:08.3278373Z 
2024-06-05T09:42:09.2972868Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0] Graph break from `Tensor.item()`, consider setting:
2024-06-05T09:42:09.2975901Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-06-05T09:42:09.2977440Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0] or:
2024-06-05T09:42:09.2978602Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-06-05T09:42:09.2980015Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0] to include these operations in the captured graph.
2024-06-05T09:42:09.2981194Z [rank0]:W0605 09:42:09.296000 139634400522880 torch/_dynamo/variables/tensor.py:715] [5/0] 
2024-06-05T09:42:20.8946112Z [rank0]:E0605 09:42:20.893000 139634400522880 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00538, (ref-fp64): 0.00555 and shape=torch.Size([4, 32001]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:42:20.8950793Z pass
2024-06-05T09:42:20.8954233Z TIMING: entire_frame_compile:36.30368 code_gen:8.896 inductor_compile:16.71377 backend_compile:27.00257
2024-06-05T09:42:20.8955670Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:10623 | FakeTensorMode.__torch_dispatch__:58178 | ProxyTorchDispatchMode.__torch_dispatch__:12396
2024-06-05T09:42:20.8956852Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique)
2024-06-05T09:42:25.5863643Z 
2024-06-05T09:42:25.8630516Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:42:25.8631197Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:42:25.8631882Z cuda eval  moondream                          
2024-06-05T09:42:25.8636114Z Traceback (most recent call last):
2024-06-05T09:42:25.8637054Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T09:42:25.8637854Z     ) = runner.load_model(
2024-06-05T09:42:25.8638638Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model
2024-06-05T09:42:25.8639506Z     benchmark = benchmark_cls(
2024-06-05T09:42:25.8640369Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T09:42:25.8641408Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T09:42:25.8642762Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__
2024-06-05T09:42:25.8644139Z     super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:42:25.8645906Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-06-05T09:42:25.8647600Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T09:42:25.8648761Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-06-05T09:42:25.8649621Z     self._skip_by_device_name()
2024-06-05T09:42:25.8650555Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-06-05T09:42:25.8651985Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-06-05T09:42:25.8653237Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`.
2024-06-05T09:42:25.8653929Z 
2024-06-05T09:42:25.8654074Z model_fail_to_load
2024-06-05T09:42:29.0584365Z 
2024-06-05T09:42:30.9337018Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2024-06-05T09:42:31.1491755Z num decayed parameter tensors: 50, with 124,354,560 parameters
2024-06-05T09:42:31.1492693Z num non-decayed parameter tensors: 98, with 121,344 parameters
2024-06-05T09:42:31.1495876Z using fused AdamW: True
2024-06-05T09:42:31.4555229Z 
2024-06-05T09:42:31.4555971Z loading model: 0it [00:02, ?it/s]
2024-06-05T09:42:31.4594506Z cuda eval  nanogpt                            
2024-06-05T09:42:44.5477296Z E0605 09:42:44.546000 140646791791232 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00437, (ref-fp64): 0.00566 and shape=torch.Size([4, 1, 50304]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:42:44.5478932Z pass
2024-06-05T09:42:44.5490445Z TIMING: code_gen:2.72515 inductor_compile:4.97763 backend_compile:9.55735 entire_frame_compile:11.51605
2024-06-05T09:42:44.5491822Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:25043 | FakeTensor.__torch_dispatch__:2657 | ProxyTorchDispatchMode.__torch_dispatch__:8284
2024-06-05T09:42:44.5493027Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique)
2024-06-05T09:42:48.3260395Z 
2024-06-05T09:42:50.6790579Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:42:50.6791087Z loading model: 0it [00:02, ?it/s]
2024-06-05T09:42:50.6794525Z cuda eval  nvidia_deeprecommender             
2024-06-05T09:42:55.2681366Z E0605 09:42:55.267000 140055370822272 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00066, (ref-fp64): 0.00062 and shape=torch.Size([4, 197951]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:42:55.2682936Z pass
2024-06-05T09:42:55.2694194Z TIMING: code_gen:0.72672 inductor_compile:0.97313 backend_compile:3.13568 entire_frame_compile:3.32889
2024-06-05T09:42:55.2695568Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1793 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170
2024-06-05T09:42:55.2696753Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique)
2024-06-05T09:42:58.7412858Z 
2024-06-05T09:42:59.6729634Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:42:59.6730421Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:42:59.6749143Z cuda eval  opacus_cifar10                     
2024-06-05T09:43:10.8604012Z E0605 09:43:10.859000 140428584698496 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00864, (ref-fp64): 0.01055 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:43:10.8605452Z pass
2024-06-05T09:43:10.8606899Z TIMING: code_gen:2.91056 inductor_compile:4.32521 backend_compile:6.6452 entire_frame_compile:9.85366
2024-06-05T09:43:10.8608468Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:7448 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866
2024-06-05T09:43:10.8609685Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-06-05T09:43:14.5829394Z 
2024-06-05T09:43:15.2089126Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:15.2089775Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:15.2154118Z cuda eval  phlippe_densenet                   
2024-06-05T09:43:29.9596302Z E0605 09:43:29.958000 140528822133376 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00077, (ref-fp64): 0.00070 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:43:29.9597642Z pass
2024-06-05T09:43:29.9609777Z TIMING: code_gen:3.95103 inductor_compile:6.84356 backend_compile:11.39255 entire_frame_compile:13.27912
2024-06-05T09:43:29.9611118Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:3838 | FakeTensorMode.__torch_dispatch__:28866 | ProxyTorchDispatchMode.__torch_dispatch__:7588
2024-06-05T09:43:29.9612300Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique)
2024-06-05T09:43:33.8829503Z 
2024-06-05T09:43:34.4161595Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:34.4162234Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:34.4186604Z cuda eval  phlippe_resnet                     
2024-06-05T09:43:41.1243788Z E0605 09:43:41.123000 139952575951488 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00838, (ref-fp64): 0.00383 and shape=torch.Size([4, 10]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:43:41.1245346Z pass
2024-06-05T09:43:41.1258517Z TIMING: code_gen:1.49388 inductor_compile:2.5498 backend_compile:4.80468 entire_frame_compile:5.44445
2024-06-05T09:43:41.1259887Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1447 | FakeTensorMode.__torch_dispatch__:10983 | ProxyTorchDispatchMode.__torch_dispatch__:2856
2024-06-05T09:43:41.1261252Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-06-05T09:43:44.6957453Z 
2024-06-05T09:43:44.9816169Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:44.9816690Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:44.9817277Z cuda eval  pyhpc_equation_of_state            
2024-06-05T09:43:51.5681361Z E0605 09:43:51.567000 140524690752128 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.03091, (ref-fp64): 1242.88896 and shape=torch.Size([4, 4, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:43:51.5683183Z pass
2024-06-05T09:43:51.5686107Z TIMING: code_gen:2.3183 inductor_compile:3.54383 backend_compile:5.05079 entire_frame_compile:5.44304
2024-06-05T09:43:51.5687782Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:10370 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061
2024-06-05T09:43:51.5688994Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique)
2024-06-05T09:43:55.0640495Z 
2024-06-05T09:43:55.3602763Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:55.3603273Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:43:55.3607410Z cuda eval  pyhpc_isoneutral_mixing            
2024-06-05T09:44:21.3757467Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 
2024-06-05T09:44:21.3758701Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in run_n_iterations
2024-06-05T09:44:21.3759929Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-06-05T09:44:21.3760859Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 430, in forward_pass
2024-06-05T09:44:21.3761875Z     return mod(*inputs)
2024-06-05T09:44:21.3763064Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T09:44:21.3763978Z     return forward_call(*args, **kwargs)
2024-06-05T09:44:21.3764997Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward
2024-06-05T09:44:21.3766078Z     return isoneutral_pytorch.isoneutral_diffusion_pre(
2024-06-05T09:44:21.3767711Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre
2024-06-05T09:44:21.3769314Z     K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :])
2024-06-05T09:44:21.3769709Z 
2024-06-05T09:44:21.4975224Z pass
2024-06-05T09:44:21.4978050Z TIMING: code_gen:5.94404 inductor_compile:15.60856 backend_compile:23.57641 entire_frame_compile:25.16352
2024-06-05T09:44:21.4979413Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:55960 | FakeTensor.__torch_dispatch__:5080 | ProxyTorchDispatchMode.__torch_dispatch__:14005
2024-06-05T09:44:21.4980605Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique)
2024-06-05T09:44:26.0182736Z 
2024-06-05T09:44:26.7585021Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-06-05T09:44:26.8407349Z 
2024-06-05T09:44:26.8408359Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:44:26.8409233Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-06-05T09:44:26.8410110Z cuda eval  pyhpc_turbulent_kinetic_energy     
2024-06-05T09:44:26.8872041Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-06-05T09:44:55.1453811Z pass
2024-06-05T09:44:55.1556598Z TIMING: code_gen:9.71943 inductor_compile:16.16013 backend_compile:25.1147 entire_frame_compile:26.85615
2024-06-05T09:44:55.1559284Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:49777 | FakeTensor.__torch_dispatch__:2561 | ProxyTorchDispatchMode.__torch_dispatch__:14635
2024-06-05T09:44:55.1560500Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique)
2024-06-05T09:44:59.5097570Z 
2024-06-05T09:45:00.2685979Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:45:00.2686903Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:45:00.2698514Z cuda eval  pytorch_CycleGAN_and_pix2pix       
2024-06-05T09:45:17.3738416Z E0605 09:45:17.373000 140039988458112 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01044, (ref-fp64): 0.01069 and shape=torch.Size([1, 3, 256, 256]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:45:17.3739965Z pass
2024-06-05T09:45:17.3773712Z TIMING: code_gen:3.80971 inductor_compile:5.8045 backend_compile:7.87059 entire_frame_compile:8.4466
2024-06-05T09:45:17.3775055Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:11393 | FakeTensor.__torch_dispatch__:1335 | ProxyTorchDispatchMode.__torch_dispatch__:3743
2024-06-05T09:45:17.3776250Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique)
2024-06-05T09:45:21.1347153Z 
2024-06-05T09:45:22.2428589Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:45:22.2429561Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:45:22.2447070Z cuda eval  pytorch_stargan                    
2024-06-05T09:45:46.1871787Z pass
2024-06-05T09:45:46.1942457Z TIMING: code_gen:2.51354 inductor_compile:4.92582 backend_compile:7.60206 entire_frame_compile:8.23946
2024-06-05T09:45:46.1943850Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:17669 | FakeTensor.__torch_dispatch__:1566 | ProxyTorchDispatchMode.__torch_dispatch__:4659
2024-06-05T09:45:46.1945060Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique)
2024-06-05T09:45:49.9432358Z 
2024-06-05T09:45:50.5782385Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:45:50.5782894Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:45:50.5803637Z cuda eval  pytorch_unet                       
2024-06-05T09:46:11.9848133Z E0605 09:46:11.983000 139940999410304 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00020 and shape=torch.Size([2, 2, 640, 959]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:46:11.9849785Z pass
2024-06-05T09:46:11.9983327Z TIMING: code_gen:4.41374 inductor_compile:6.44373 backend_compile:8.55118 entire_frame_compile:9.40241
2024-06-05T09:46:11.9984941Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1807 | FakeTensorMode.__torch_dispatch__:14774 | ProxyTorchDispatchMode.__torch_dispatch__:4409
2024-06-05T09:46:11.9986154Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-06-05T09:46:15.7013820Z 
2024-06-05T09:46:16.5377649Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth
2024-06-05T09:46:16.5552149Z 
2024-06-05T09:46:16.5552379Z 
2024-06-05T09:46:16.6556337Z   0% 0.00/230M [00:00<?, ?B/s]
2024-06-05T09:46:16.6556715Z 
2024-06-05T09:46:16.7578504Z  20% 46.9M/230M [00:00<00:00, 491MB/s]
2024-06-05T09:46:16.7578882Z 
2024-06-05T09:46:16.8583883Z  41% 93.8M/230M [00:00<00:00, 485MB/s]
2024-06-05T09:46:16.8584268Z 
2024-06-05T09:46:16.9588998Z  61% 140M/230M [00:00<00:00, 484MB/s] 
2024-06-05T09:46:16.9589628Z 
2024-06-05T09:46:17.0609091Z  81% 186M/230M [00:00<00:00, 484MB/s]
2024-06-05T09:46:17.0609628Z 100% 230M/230M [00:00<00:00, 478MB/s]
2024-06-05T09:46:18.0045899Z 
2024-06-05T09:46:18.0046884Z loading model: 0it [00:02, ?it/s]
2024-06-05T09:46:18.0223728Z cuda eval  resnet152                          
2024-06-05T09:46:49.9837247Z E0605 09:46:49.982000 139760882430592 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01578, (ref-fp64): 0.01583 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:46:49.9838837Z pass
2024-06-05T09:46:49.9862463Z TIMING: code_gen:4.74731 inductor_compile:12.86944 backend_compile:24.5581 entire_frame_compile:28.55188
2024-06-05T09:46:49.9863839Z STATS: call_* op count: 1030 | FakeTensor.__torch_dispatch__:11193 | FakeTensorMode.__torch_dispatch__:80746 | ProxyTorchDispatchMode.__torch_dispatch__:20906
2024-06-05T09:46:49.9865083Z Dynamo produced 1 graphs covering 1030 ops with 0 graph breaks (0 unique)
2024-06-05T09:46:54.6023687Z 
2024-06-05T09:46:54.9338069Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
2024-06-05T09:46:54.9501135Z 
2024-06-05T09:46:54.9501366Z 
2024-06-05T09:46:55.0504413Z   0% 0.00/44.7M [00:00<?, ?B/s]
2024-06-05T09:46:55.0504757Z 
2024-06-05T09:46:55.0593564Z  91% 40.6M/44.7M [00:00<00:00, 426MB/s]
2024-06-05T09:46:55.0594057Z 100% 44.7M/44.7M [00:00<00:00, 429MB/s]
2024-06-05T09:46:55.6941636Z 
2024-06-05T09:46:55.6942189Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:46:55.6967394Z cuda eval  resnet18                           
2024-06-05T09:47:03.2455033Z E0605 09:47:03.244000 140114825339520 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00580, (ref-fp64): 0.00713 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:47:03.2456719Z pass
2024-06-05T09:47:03.2475734Z TIMING: code_gen:1.71356 inductor_compile:2.84637 backend_compile:5.18143 entire_frame_compile:5.81994
2024-06-05T09:47:03.2477155Z STATS: call_* op count: 138 | FakeTensor.__torch_dispatch__:1475 | FakeTensorMode.__torch_dispatch__:11021 | ProxyTorchDispatchMode.__torch_dispatch__:2787
2024-06-05T09:47:03.2478337Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-06-05T09:47:06.9067805Z 
2024-06-05T09:47:07.3737012Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
2024-06-05T09:47:07.3905807Z 
2024-06-05T09:47:07.3906108Z 
2024-06-05T09:47:07.4909866Z   0% 0.00/97.8M [00:00<?, ?B/s]
2024-06-05T09:47:07.4910274Z 
2024-06-05T09:47:07.6043805Z  42% 41.4M/97.8M [00:00<00:00, 433MB/s]
2024-06-05T09:47:07.6044178Z 
2024-06-05T09:47:07.6621031Z  85% 82.8M/97.8M [00:00<00:00, 402MB/s]
2024-06-05T09:47:07.6621561Z 100% 97.8M/97.8M [00:00<00:00, 378MB/s]
2024-06-05T09:47:08.3903555Z 
2024-06-05T09:47:08.3904405Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:47:08.3966719Z cuda eval  resnet50                           
2024-06-05T09:47:21.9917819Z E0605 09:47:21.990000 140344353399424 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00836, (ref-fp64): 0.00867 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:47:21.9919467Z pass
2024-06-05T09:47:21.9944124Z TIMING: code_gen:2.27418 inductor_compile:5.22422 backend_compile:9.51389 entire_frame_compile:10.92789
2024-06-05T09:47:21.9945561Z STATS: call_* op count: 350 | FakeTensor.__torch_dispatch__:3849 | FakeTensorMode.__torch_dispatch__:28114 | ProxyTorchDispatchMode.__torch_dispatch__:7204
2024-06-05T09:47:21.9946784Z Dynamo produced 1 graphs covering 350 ops with 0 graph breaks (0 unique)
2024-06-05T09:47:25.8129360Z 
2024-06-05T09:47:26.0188392Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:47:26.0189341Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:47:26.0189846Z cuda eval  resnet50_quantized_qat             
2024-06-05T09:47:26.0194400Z Traceback (most recent call last):
2024-06-05T09:47:26.0195237Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T09:47:26.0197907Z     ) = runner.load_model(
2024-06-05T09:47:26.0198857Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 310, in load_model
2024-06-05T09:47:26.0199646Z     benchmark = benchmark_cls(
2024-06-05T09:47:26.0200432Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T09:47:26.0201464Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T09:47:26.0202955Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat/__init__.py", line 21, in __init__
2024-06-05T09:47:26.0204085Z     raise NotImplementedError("The eval test only supports CPU.")
2024-06-05T09:47:26.0204783Z NotImplementedError: The eval test only supports CPU.
2024-06-05T09:47:26.0205180Z 
2024-06-05T09:47:26.0205316Z model_fail_to_load
2024-06-05T09:47:29.1794811Z 
2024-06-05T09:47:29.6514345Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnext50_32x4d-7cdf4587.pth
2024-06-05T09:47:29.6677389Z 
2024-06-05T09:47:29.6678021Z 
2024-06-05T09:47:29.7687769Z   0% 0.00/95.8M [00:00<?, ?B/s]
2024-06-05T09:47:29.7688096Z 
2024-06-05T09:47:29.8768377Z  16% 15.0M/95.8M [00:00<00:00, 156MB/s]
2024-06-05T09:47:29.8769471Z 
2024-06-05T09:47:29.9769701Z  31% 29.9M/95.8M [00:00<00:00, 149MB/s]
2024-06-05T09:47:29.9770066Z 
2024-06-05T09:47:30.0291238Z  76% 72.9M/95.8M [00:00<00:00, 283MB/s]
2024-06-05T09:47:30.0291738Z 100% 95.8M/95.8M [00:00<00:00, 278MB/s]
2024-06-05T09:47:30.8247594Z 
2024-06-05T09:47:30.8248287Z loading model: 0it [00:01, ?it/s]
2024-06-05T09:47:30.8308727Z cuda eval  resnext50_32x4d                    
2024-06-05T09:47:45.4297152Z E0605 09:47:45.428000 140123416060544 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01192, (ref-fp64): 0.01169 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:47:45.4298561Z pass
2024-06-05T09:47:45.4323431Z TIMING: code_gen:2.35278 inductor_compile:5.30383 backend_compile:9.6051 entire_frame_compile:11.02002
2024-06-05T09:47:45.4324826Z STATS: call_* op count: 350 | FakeTensor.__torch_dispatch__:3847 | FakeTensorMode.__torch_dispatch__:28114 | ProxyTorchDispatchMode.__torch_dispatch__:7204
2024-06-05T09:47:45.4326023Z Dynamo produced 1 graphs covering 350 ops with 0 graph breaks (0 unique)
2024-06-05T09:47:49.3345127Z 
2024-06-05T09:47:55.2590719Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:47:55.2591409Z loading model: 0it [00:05, ?it/s]
2024-06-05T09:47:55.2709326Z cuda eval  sam                                
2024-06-05T09:49:43.5888008Z E0605 09:49:43.587000 140064104473216 torch/_dynamo/utils.py:1482] key: iou_predictions, passes_test: True, RMSE (res-fp64): 0.00095, (ref-fp64): 0.00295 and shape=torch.Size([1, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:49:43.5890925Z E0605 09:49:43.588000 140064104473216 torch/_dynamo/utils.py:1482] key: low_res_logits, passes_test: True, RMSE (res-fp64): 0.05577, (ref-fp64): 0.16084 and shape=torch.Size([1, 1, 256, 256]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:49:43.5895873Z pass
2024-06-05T09:49:43.5995905Z TIMING: code_gen:16.29704 inductor_compile:31.16725 backend_compile:61.93299 entire_frame_compile:70.15171
2024-06-05T09:49:43.5997281Z STATS: call_* op count: 4569 | FakeTensorMode.__torch_dispatch__:171557 | FakeTensor.__torch_dispatch__:12878 | ProxyTorchDispatchMode.__torch_dispatch__:55990
2024-06-05T09:49:43.5998486Z Dynamo produced 1 graphs covering 4569 ops with 0 graph breaks (0 unique)
2024-06-05T09:49:49.4949011Z 
2024-06-05T09:51:23.4623860Z loading model: 0it [00:00, ?it/s]W0605 09:51:23.461000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:23.8412898Z W0605 09:51:23.840000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:24.4618236Z W0605 09:51:24.461000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:24.4623761Z AUTOTUNE mm(4096x1280, 1280x5120)
2024-06-05T09:51:24.4624220Z   triton_mm_93 0.7885 ms 100.0%
2024-06-05T09:51:24.4624635Z   triton_mm_86 0.7895 ms 99.9%
2024-06-05T09:51:24.4625047Z   triton_mm_90 0.7895 ms 99.9%
2024-06-05T09:51:24.4625453Z   triton_mm_88 0.8008 ms 98.5%
2024-06-05T09:51:24.4625881Z   triton_mm_83 0.8038 ms 98.1%
2024-06-05T09:51:24.4626283Z   triton_mm_91 0.8120 ms 97.1%
2024-06-05T09:51:24.4626699Z   triton_mm_84 0.8151 ms 96.7%
2024-06-05T09:51:24.4627100Z   triton_mm_87 0.8192 ms 96.3%
2024-06-05T09:51:24.4628727Z   triton_mm_94 0.8325 ms 94.7%
2024-06-05T09:51:24.4629250Z   mm 0.8407 ms 93.8%
2024-06-05T09:51:24.4629914Z SingleProcess AUTOTUNE benchmarking takes 2.1398 seconds and 0.0016 seconds precompiling
2024-06-05T09:51:27.1807542Z W0605 09:51:27.180000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:27.5527060Z W0605 09:51:27.551000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:28.1732416Z W0605 09:51:28.172000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:28.1737939Z AUTOTUNE mm(4096x5120, 5120x1280)
2024-06-05T09:51:28.1738409Z   triton_mm_3474 0.7803 ms 100.0%
2024-06-05T09:51:28.1738846Z   triton_mm_3476 0.7803 ms 100.0%
2024-06-05T09:51:28.1739272Z   triton_mm_3473 0.7844 ms 99.5%
2024-06-05T09:51:28.1739698Z   triton_mm_3469 0.7864 ms 99.2%
2024-06-05T09:51:28.1740133Z   triton_mm_3470 0.7875 ms 99.1%
2024-06-05T09:51:28.1740539Z   mm 0.7895 ms 98.8%
2024-06-05T09:51:28.1740904Z   triton_mm_3471 0.7905 ms 98.7%
2024-06-05T09:51:28.1741320Z   triton_mm_3477 0.7905 ms 98.7%
2024-06-05T09:51:28.1741737Z   triton_mm_3467 0.8151 ms 95.7%
2024-06-05T09:51:28.1742140Z   triton_mm_3466 0.8192 ms 95.3%
2024-06-05T09:51:28.1742871Z SingleProcess AUTOTUNE benchmarking takes 2.1373 seconds and 0.0018 seconds precompiling
2024-06-05T09:51:31.3209052Z AUTOTUNE convolution(1x3x1024x1024, 1280x3x16x16)
2024-06-05T09:51:31.3209737Z   triton_convolution_3 0.8632 ms 100.0%
2024-06-05T09:51:31.3210829Z   triton_convolution_6 1.0527 ms 82.0%
2024-06-05T09:51:31.3211406Z   triton_convolution_1 1.1069 ms 78.0%
2024-06-05T09:51:31.3211881Z   triton_convolution_0 1.2125 ms 71.2%
2024-06-05T09:51:31.3212355Z   convolution 1.4746 ms 58.5%
2024-06-05T09:51:31.3212850Z   triton_convolution_5 1.6312 ms 52.9%
2024-06-05T09:51:31.3213457Z   triton_convolution_4 1.6773 ms 51.5%
2024-06-05T09:51:31.3214087Z   triton_convolution_2 4.1216 ms 20.9%
2024-06-05T09:51:31.3215047Z SingleProcess AUTOTUNE benchmarking takes 1.1210 seconds and 0.0006 seconds precompiling
2024-06-05T09:51:32.4533262Z W0605 09:51:32.452000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:32.8213970Z W0605 09:51:32.820000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:33.4343515Z W0605 09:51:33.433000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:33.4349625Z AUTOTUNE mm(4904x1280, 1280x3840)
2024-06-05T09:51:33.4350261Z   triton_mm_16 0.7168 ms 100.0%
2024-06-05T09:51:33.4350847Z   triton_mm_18 0.7229 ms 99.2%
2024-06-05T09:51:33.4351418Z   triton_mm_13 0.7281 ms 98.5%
2024-06-05T09:51:33.4351846Z   triton_mm_20 0.7342 ms 97.6%
2024-06-05T09:51:33.4352268Z   triton_mm_14 0.7373 ms 97.2%
2024-06-05T09:51:33.4352669Z   triton_mm_23 0.7393 ms 97.0%
2024-06-05T09:51:33.4353067Z   triton_mm_17 0.7424 ms 96.6%
2024-06-05T09:51:33.4353470Z   triton_mm_21 0.7537 ms 95.1%
2024-06-05T09:51:33.4353873Z   triton_mm_24 0.7741 ms 92.6%
2024-06-05T09:51:33.4354279Z   triton_mm_12 0.7936 ms 90.3%
2024-06-05T09:51:33.4354970Z SingleProcess AUTOTUNE benchmarking takes 2.1128 seconds and 0.0013 seconds precompiling
2024-06-05T09:51:35.2365102Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2024-06-05T09:51:35.2365775Z   triton_bmm_27 0.0369 ms 100.0%
2024-06-05T09:51:35.2366355Z   triton_bmm_30 0.0369 ms 100.0%
2024-06-05T09:51:35.2367909Z   triton_bmm_33 0.0369 ms 100.0%
2024-06-05T09:51:35.2368478Z   triton_bmm_36 0.0369 ms 100.0%
2024-06-05T09:51:35.2368948Z   triton_bmm_40 0.0369 ms 100.0%
2024-06-05T09:51:35.2369557Z   triton_bmm_41 0.0379 ms 97.3%
2024-06-05T09:51:35.2370060Z   triton_bmm_28 0.0399 ms 92.3%
2024-06-05T09:51:35.2370471Z   triton_bmm_29 0.0399 ms 92.3%
2024-06-05T09:51:35.2370879Z   triton_bmm_32 0.0410 ms 90.0%
2024-06-05T09:51:35.2371278Z   triton_bmm_35 0.0410 ms 90.0%
2024-06-05T09:51:35.2372069Z SingleProcess AUTOTUNE benchmarking takes 1.7999 seconds and 0.0012 seconds precompiling
2024-06-05T09:51:37.0397224Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2024-06-05T09:51:37.0397781Z   triton_bmm_46 0.0410 ms 100.0%
2024-06-05T09:51:37.0398382Z   triton_bmm_49 0.0410 ms 100.0%
2024-06-05T09:51:37.0398976Z   triton_bmm_52 0.0410 ms 100.0%
2024-06-05T09:51:37.0399420Z   triton_bmm_43 0.0420 ms 97.6%
2024-06-05T09:51:37.0399837Z   triton_bmm_56 0.0420 ms 97.6%
2024-06-05T09:51:37.0400249Z   triton_bmm_57 0.0420 ms 97.6%
2024-06-05T09:51:37.0400662Z   triton_bmm_44 0.0451 ms 90.9%
2024-06-05T09:51:37.0401077Z   triton_bmm_45 0.0451 ms 90.9%
2024-06-05T09:51:37.0401470Z   bmm 0.0461 ms 88.9%
2024-06-05T09:51:37.0401838Z   triton_bmm_51 0.0461 ms 88.9%
2024-06-05T09:51:37.0402741Z SingleProcess AUTOTUNE benchmarking takes 1.8013 seconds and 0.0017 seconds precompiling
2024-06-05T09:51:38.0823484Z W0605 09:51:38.081000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:38.4289929Z W0605 09:51:38.428000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:38.9991273Z W0605 09:51:38.998000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:38.9998375Z AUTOTUNE mm(4900x1280, 1280x1280)
2024-06-05T09:51:38.9999027Z   triton_mm_67 0.2499 ms 100.0%
2024-06-05T09:51:38.9999653Z   triton_mm_71 0.2499 ms 100.0%
2024-06-05T09:51:39.0000251Z   triton_mm_64 0.2519 ms 99.2%
2024-06-05T09:51:39.0000810Z   triton_mm_74 0.2529 ms 98.8%
2024-06-05T09:51:39.0001388Z   triton_mm_69 0.2549 ms 98.0%
2024-06-05T09:51:39.0002053Z   triton_mm_65 0.2550 ms 98.0%
2024-06-05T09:51:39.0002465Z   triton_mm_72 0.2580 ms 96.8%
2024-06-05T09:51:39.0002940Z   triton_mm_68 0.2591 ms 96.4%
2024-06-05T09:51:39.0003348Z   triton_mm_75 0.2642 ms 94.6%
2024-06-05T09:51:39.0003823Z   triton_mm_63 0.2673 ms 93.5%
2024-06-05T09:51:39.0004798Z SingleProcess AUTOTUNE benchmarking takes 1.9585 seconds and 0.0016 seconds precompiling
2024-06-05T09:51:40.2569735Z W0605 09:51:40.256000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:40.6190415Z W0605 09:51:40.618000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:41.2277466Z W0605 09:51:41.227000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:41.2286780Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2024-06-05T09:51:41.2288157Z   triton_mm_776 0.5919 ms 100.0%
2024-06-05T09:51:41.2288735Z   triton_mm_779 0.5949 ms 99.5%
2024-06-05T09:51:41.2289227Z   triton_mm_772 0.5970 ms 99.1%
2024-06-05T09:51:41.2289778Z   triton_mm_774 0.6042 ms 98.0%
2024-06-05T09:51:41.2290353Z   triton_mm_769 0.6072 ms 97.5%
2024-06-05T09:51:41.2290786Z   triton_mm_777 0.6083 ms 97.3%
2024-06-05T09:51:41.2291192Z   triton_mm_770 0.6216 ms 95.2%
2024-06-05T09:51:41.2291632Z   triton_mm_773 0.6216 ms 95.2%
2024-06-05T09:51:41.2292044Z   triton_mm_780 0.6359 ms 93.1%
2024-06-05T09:51:41.2292450Z   bias_addmm 0.6420 ms 92.2%
2024-06-05T09:51:41.2293129Z SingleProcess AUTOTUNE benchmarking takes 2.2202 seconds and 0.0015 seconds precompiling
2024-06-05T09:51:43.2493789Z AUTOTUNE bmm(64x1024x80, 64x80x64)
2024-06-05T09:51:43.2494510Z   triton_bmm_793 0.0440 ms 100.0%
2024-06-05T09:51:43.2495103Z   bmm 0.0442 ms 99.6%
2024-06-05T09:51:43.2495588Z   triton_bmm_783 0.0451 ms 97.7%
2024-06-05T09:51:43.2496166Z   triton_bmm_786 0.0451 ms 97.7%
2024-06-05T09:51:43.2496681Z   triton_bmm_789 0.0451 ms 97.7%
2024-06-05T09:51:43.2497231Z   triton_bmm_784 0.0461 ms 95.6%
2024-06-05T09:51:43.2497838Z   triton_bmm_785 0.0461 ms 95.6%
2024-06-05T09:51:43.2498418Z   triton_bmm_790 0.0471 ms 93.5%
2024-06-05T09:51:43.2499016Z   triton_bmm_792 0.0471 ms 93.5%
2024-06-05T09:51:43.2499590Z   triton_bmm_794 0.0471 ms 93.5%
2024-06-05T09:51:43.2500503Z SingleProcess AUTOTUNE benchmarking takes 2.0191 seconds and 0.0013 seconds precompiling
2024-06-05T09:51:44.2814557Z W0605 09:51:44.280000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:44.6191865Z W0605 09:51:44.618000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:45.1816107Z W0605 09:51:45.181000 139781579743872 torch/_inductor/select_algorithm.py:1467] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-06-05T09:51:45.1825310Z AUTOTUNE mm(4096x1280, 1280x1280)
2024-06-05T09:51:45.1826253Z   triton_mm_827 0.2017 ms 100.0%
2024-06-05T09:51:45.1826728Z   triton_mm_831 0.2017 ms 100.0%
2024-06-05T09:51:45.1827154Z   triton_mm_824 0.2038 ms 99.0%
2024-06-05T09:51:45.1827575Z   triton_mm_834 0.2048 ms 98.5%
2024-06-05T09:51:45.1827994Z   triton_mm_829 0.2058 ms 98.0%
2024-06-05T09:51:45.1828406Z   triton_mm_832 0.2088 ms 96.6%
2024-06-05T09:51:45.1828820Z   triton_mm_828 0.2099 ms 96.1%
2024-06-05T09:51:45.1829235Z   triton_mm_825 0.2120 ms 95.2%
2024-06-05T09:51:45.1829626Z   mm 0.2140 ms 94.3%
2024-06-05T09:51:45.1829990Z   triton_mm_835 0.2140 ms 94.3%
2024-06-05T09:51:45.1830678Z SingleProcess AUTOTUNE benchmarking takes 1.9314 seconds and 0.0013 seconds precompiling
2024-06-05T09:51:46.2031284Z AUTOTUNE convolution(1x1280x64x64, 256x1280x1x1)
2024-06-05T09:51:46.2032221Z   convolution 0.0574 ms 100.0%
2024-06-05T09:51:46.2032697Z   triton_convolution_3483 0.0625 ms 91.9%
2024-06-05T09:51:46.2033464Z   triton_convolution_3482 0.0696 ms 82.4%
2024-06-05T09:51:46.2033971Z   triton_convolution_3484 0.0717 ms 80.0%
2024-06-05T09:51:46.2034443Z   conv1x1_via_mm 0.0788 ms 72.8%
2024-06-05T09:51:46.2034882Z   triton_convolution_3485 0.0860 ms 66.7%
2024-06-05T09:51:46.2035364Z   triton_convolution_3480 0.1004 ms 57.2%
2024-06-05T09:51:46.2035845Z   triton_convolution_3479 0.1106 ms 51.9%
2024-06-05T09:51:46.2036323Z   triton_convolution_3481 0.1599 ms 35.9%
2024-06-05T09:51:46.2037055Z SingleProcess AUTOTUNE benchmarking takes 0.9894 seconds and 0.0005 seconds precompiling
2024-06-05T09:51:47.1236384Z AUTOTUNE convolution(1x256x64x64, 256x256x3x3)
2024-06-05T09:51:47.1237052Z   convolution 0.1198 ms 100.0%
2024-06-05T09:51:47.1237486Z   triton_convolution_3492 0.2212 ms 54.2%
2024-06-05T09:51:47.1237973Z   triton_convolution_3487 0.2570 ms 46.6%
2024-06-05T09:51:47.1238453Z   triton_convolution_3489 0.2918 ms 41.1%
2024-06-05T09:51:47.1238933Z   triton_convolution_3488 0.3441 ms 34.8%
2024-06-05T09:51:47.1239419Z   triton_convolution_3490 0.3604 ms 33.2%
2024-06-05T09:51:47.1239907Z   triton_convolution_3491 0.4076 ms 29.4%
2024-06-05T09:51:47.1240384Z   triton_convolution_3486 0.5878 ms 20.4%
2024-06-05T09:51:47.1241113Z SingleProcess AUTOTUNE benchmarking takes 0.9188 seconds and 0.0005 seconds precompiling
2024-06-05T09:52:03.0910234Z Warning: Custom flash attention kernels were written specifically for A100.
2024-06-05T09:52:03.0915799Z We will try to read previously created kernel configurations from /var/lib/jenkins/workspace/flash_4_configs.p.
2024-06-05T09:52:03.0917751Z You can disable this kernel by setting SEGMENT_ANYTHING_FAST_USE_FLASH_4=0
2024-06-05T09:52:03.0919640Z key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))  not found. Running autotune. This might take a while.
2024-06-05T09:52:03.0921274Z all configs len:  60
2024-06-05T09:52:07.9663940Z (64, 64, 1, 1)  : 28828.37219949579
2024-06-05T09:52:10.4275239Z (64, 64, 2, 1)  : 15728.520800621482
2024-06-05T09:52:14.2496687Z (64, 64, 2, 2)  : 8552.682139998069
2024-06-05T09:52:16.0034713Z (64, 64, 4, 1)  : 3855.5368600646034
2024-06-05T09:52:17.7272944Z (64, 64, 4, 2)  : 3648.405800049659
2024-06-05T09:52:19.3767173Z (64, 64, 4, 3)  : 3103.104350011563
2024-06-05T09:52:20.4399318Z (64, 64, 4, 4)  : None
2024-06-05T09:52:21.8889786Z (64, 64, 8, 1)  : 4008.0131299328064
2024-06-05T09:52:23.3413454Z (64, 64, 8, 2)  : 3925.0643800187395
2024-06-05T09:52:24.6729816Z (64, 64, 8, 3)  : 3339.1778900113422
2024-06-05T09:52:25.3258783Z (64, 64, 8, 4)  : None
2024-06-05T09:52:25.9946691Z (64, 64, 8, 5)  : None
2024-06-05T09:52:26.6823356Z (64, 64, 8, 6)  : None
2024-06-05T09:52:27.3949551Z (64, 64, 8, 7)  : None
2024-06-05T09:52:28.1272589Z (64, 64, 8, 8)  : None
2024-06-05T09:52:37.1495889Z (64, 128, 1, 1)  : 67052.03169985907
2024-06-05T09:52:41.2519386Z (64, 128, 2, 1)  : 18777.18000032473
2024-06-05T09:52:45.1870752Z (64, 128, 2, 2)  : 20106.560399290174
2024-06-05T09:52:47.4186585Z (64, 128, 4, 1)  : 3845.706279971637
2024-06-05T09:52:49.7152494Z (64, 128, 4, 2)  : 4239.183769968804
2024-06-05T09:52:51.1976212Z (64, 128, 4, 3)  : None
2024-06-05T09:52:52.8078887Z (64, 128, 4, 4)  : None
2024-06-05T09:52:54.3877482Z (64, 128, 8, 1)  : 3522.667919896776
2024-06-05T09:52:55.9336636Z (64, 128, 8, 2)  : 3387.9934999276884
2024-06-05T09:52:56.8075370Z (64, 128, 8, 3)  : None
2024-06-05T09:52:57.7211974Z (64, 128, 8, 4)  : None
2024-06-05T09:52:58.6754365Z (64, 128, 8, 5)  : None
2024-06-05T09:52:59.7027148Z (64, 128, 8, 6)  : None
2024-06-05T09:53:00.7656703Z (64, 128, 8, 7)  : None
2024-06-05T09:53:01.9010209Z (64, 128, 8, 8)  : None
2024-06-05T09:53:15.3225646Z (128, 64, 1, 1)  : 81804.99170121038
2024-06-05T09:53:21.3236909Z (128, 64, 2, 1)  : 77490.06560043199
2024-06-05T09:53:26.1494683Z (128, 64, 2, 2)  : 26127.52150016604
2024-06-05T09:53:29.4556322Z (128, 64, 4, 1)  : 6905.0693200551905
2024-06-05T09:53:32.3773977Z (128, 64, 4, 2)  : 4992.128320009215
2024-06-05T09:53:35.5216805Z (128, 64, 4, 3)  : 5731.283490022179
2024-06-05T09:53:37.4974844Z (128, 64, 4, 4)  : None
2024-06-05T09:53:38.9827163Z (128, 64, 8, 1)  : 2846.4450399042107
2024-06-05T09:53:40.4463763Z (128, 64, 8, 2)  : 2655.0940399465617
2024-06-05T09:53:41.4274299Z (128, 64, 8, 3)  : None
2024-06-05T09:53:42.4079966Z (128, 64, 8, 4)  : None
2024-06-05T09:53:43.4006636Z (128, 64, 8, 5)  : None
2024-06-05T09:53:44.4210967Z (128, 64, 8, 6)  : None
2024-06-05T09:53:45.4549525Z (128, 64, 8, 7)  : None
2024-06-05T09:53:46.5115333Z (128, 64, 8, 8)  : None
2024-06-05T09:54:08.2776910Z (128, 128, 1, 1)  : 93159.32999985914
2024-06-05T09:54:16.4109214Z (128, 128, 2, 1)  : 100455.86399792228
2024-06-05T09:54:24.5148386Z (128, 128, 2, 2)  : 90838.74467372273
2024-06-05T09:54:27.9052399Z (128, 128, 4, 1)  : 10789.575050148414
2024-06-05T09:54:31.3556064Z (128, 128, 4, 2)  : 9972.420733538456
2024-06-05T09:54:34.4918774Z (128, 128, 4, 3)  : None
2024-06-05T09:54:37.7525706Z (128, 128, 4, 4)  : None
2024-06-05T09:54:39.1515412Z (128, 128, 8, 1)  : None
2024-06-05T09:54:40.5477980Z (128, 128, 8, 2)  : None
2024-06-05T09:54:42.0086072Z (128, 128, 8, 3)  : None
2024-06-05T09:54:43.5116817Z (128, 128, 8, 4)  : None
2024-06-05T09:54:45.0654072Z (128, 128, 8, 5)  : None
2024-06-05T09:54:46.6823933Z (128, 128, 8, 6)  : None
2024-06-05T09:54:48.3364976Z (128, 128, 8, 7)  : None
2024-06-05T09:54:50.0349478Z (128, 128, 8, 8)  : None
2024-06-05T09:54:50.0351404Z Found best_config  (128, 64, 8, 2)  with time  2655.0940399465617  for key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))
2024-06-05T09:54:50.0353390Z Warning: Custom flash attention kernels were written specifically for A100.
2024-06-05T09:54:50.0354361Z Storing configs for NVIDIA A10G locally under /var/lib/jenkins/workspace/flash_4_configs.p
2024-06-05T09:54:50.0355285Z Saving best configs to file /var/lib/jenkins/workspace/flash_4_configs.p
2024-06-05T09:54:51.5668320Z 
2024-06-05T09:54:51.5668628Z loading model: 0it [05:02, ?it/s]
2024-06-05T09:54:51.5718373Z cuda eval  sam_fast                           
2024-06-05T09:55:15.6063900Z W0605 09:55:15.605000 139781579743872 torch/fx/experimental/symbolic_shapes.py:4424] [0/1] r0 is not in var_ranges, defaulting to unknown range.
2024-06-05T09:55:21.8208606Z W0605 09:55:21.820000 139781579743872 torch/fx/experimental/symbolic_shapes.py:4424] [0/1] q0 is not in var_ranges, defaulting to unknown range.
2024-06-05T09:55:21.8687361Z W0605 09:55:21.868000 139781579743872 torch/fx/experimental/symbolic_shapes.py:4424] [0/1] z0 is not in var_ranges, defaulting to unknown range.
2024-06-05T09:55:28.0419592Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2024-06-05T09:55:28.0420184Z   bias_addmm 85.9351 ms 100.0%
2024-06-05T09:55:28.0421186Z   addmm 86.2740 ms 99.6%
2024-06-05T09:55:28.0422715Z SingleProcess AUTOTUNE benchmarking takes 1.4083 seconds and 0.0000 seconds precompiling
2024-06-05T09:55:28.2695805Z W0605 09:55:28.268000 139781579743872 torch/fx/experimental/symbolic_shapes.py:4424] [0/1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T09:59:19.5401034Z E0605 09:59:19.539000 139781579743872 torch/_dynamo/utils.py:1482] key: low_res_logits, passes_test: True, RMSE (res-fp64): 0.04625, (ref-fp64): 0.68421 and shape=torch.Size([1, 1, 256, 256]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T09:59:19.5403632Z pass
2024-06-05T09:59:19.5447916Z TIMING: code_gen:14.63363 inductor_compile:31.06976 backend_compile:59.84282 entire_frame_compile:68.34455
2024-06-05T09:59:19.5449686Z STATS: call_* op count: 13017 | FakeTensor.__torch_dispatch__:57700 | FakeTensorMode.__torch_dispatch__:555136 | ProxyTorchDispatchMode.__torch_dispatch__:163630
2024-06-05T09:59:19.5450926Z Dynamo produced 4 graphs covering 10931 ops with 0 graph breaks (0 unique)
2024-06-05T09:59:32.8355716Z 
2024-06-05T09:59:33.0699140Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth
2024-06-05T09:59:33.0865751Z 
2024-06-05T09:59:33.0865874Z 
2024-06-05T09:59:33.1118384Z   0% 0.00/8.79M [00:00<?, ?B/s]
2024-06-05T09:59:33.1119078Z 100% 8.79M/8.79M [00:00<00:00, 370MB/s]
2024-06-05T09:59:33.7167134Z 
2024-06-05T09:59:33.7168186Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:59:33.7233326Z cuda eval  shufflenet_v2_x1_0                 
2024-06-05T09:59:49.1207179Z E0605 09:59:49.119000 139991056016000 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01915, (ref-fp64): 0.01937 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:59:49.1208674Z pass
2024-06-05T09:59:49.1228540Z TIMING: code_gen:3.64834 inductor_compile:6.93011 backend_compile:12.20374 entire_frame_compile:13.83264
2024-06-05T09:59:49.1229894Z STATS: call_* op count: 542 | FakeTensor.__torch_dispatch__:4080 | FakeTensorMode.__torch_dispatch__:32437 | ProxyTorchDispatchMode.__torch_dispatch__:8389
2024-06-05T09:59:49.1231087Z Dynamo produced 1 graphs covering 542 ops with 0 graph breaks (0 unique)
2024-06-05T09:59:53.0521357Z 
2024-06-05T09:59:53.5688906Z loading model: 0it [00:00, ?it/s]/opt/conda/envs/py_3.10/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
2024-06-05T09:59:53.5690554Z   if not isinstance(terminated, (bool, np.bool8)):
2024-06-05T09:59:53.8355294Z 
2024-06-05T09:59:53.8355985Z loading model: 0it [00:00, ?it/s]
2024-06-05T09:59:53.8358481Z cuda eval  soft_actor_critic                  
2024-06-05T09:59:57.1408391Z E0605 09:59:57.140000 139966919492224 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00002, (ref-fp64): 0.00022 and shape=torch.Size([256, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:59:57.1413898Z E0605 09:59:57.140000 139966919492224 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00002, (ref-fp64): 0.00022 and shape=torch.Size([256, 1]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T09:59:57.1417333Z pass
2024-06-05T09:59:57.1432333Z TIMING: code_gen:0.62153 inductor_compile:0.88679 backend_compile:1.99094 entire_frame_compile:2.16303
2024-06-05T09:59:57.1433670Z STATS: call_* op count: 32 | FakeTensor.__torch_dispatch__:127 | FakeTensorMode.__torch_dispatch__:1328 | ProxyTorchDispatchMode.__torch_dispatch__:283
2024-06-05T09:59:57.1434861Z Dynamo produced 1 graphs covering 32 ops with 0 graph breaks (0 unique)
2024-06-05T10:00:00.6525744Z 
2024-06-05T10:00:02.0230581Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:00:02.0231563Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:00:02.0291973Z cuda eval  speech_transformer                 
2024-06-05T10:00:12.0067760Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg103_1). Found from : 
2024-06-05T10:00:12.0069591Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 58, in torch_dynamo_resume_in_forward_at_50
2024-06-05T10:00:12.0071010Z     enc_output, enc_slf_attn = enc_layer(
2024-06-05T10:00:12.0072230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:12.0073190Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:12.0075470Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 84, in forward
2024-06-05T10:00:12.0076964Z     enc_output, enc_slf_attn = self.slf_attn(
2024-06-05T10:00:12.0078358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:12.0079288Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:12.0080451Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-06-05T10:00:12.0081659Z     output, attn = self.attention(q, k, v, mask=mask)
2024-06-05T10:00:12.0082815Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:12.0083705Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:12.0084875Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-06-05T10:00:12.0086031Z     attn = attn / self.temperature
2024-06-05T10:00:12.0086327Z 
2024-06-05T10:00:25.3920295Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg162_1). Found from : 
2024-06-05T10:00:25.3922554Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 105, in torch_dynamo_resume_in_forward_at_96
2024-06-05T10:00:25.3923960Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2024-06-05T10:00:25.3925136Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:25.3926053Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:25.3927402Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 243, in forward
2024-06-05T10:00:25.3928586Z     dec_output, dec_slf_attn = self.slf_attn(
2024-06-05T10:00:25.3929592Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:25.3930500Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:25.3931723Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-06-05T10:00:25.3932948Z     output, attn = self.attention(q, k, v, mask=mask)
2024-06-05T10:00:25.3933987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:00:25.3934876Z     return forward_call(*args, **kwargs)
2024-06-05T10:00:25.3936040Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-06-05T10:00:25.3937186Z     attn = attn / self.temperature
2024-06-05T10:00:25.3937487Z 
2024-06-05T10:00:26.7592135Z E0605 10:00:26.758000 139868493980288 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00371, (ref-fp64): 0.00386 and shape=torch.Size([10, 22, 1014]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:00:26.7595746Z pass
2024-06-05T10:00:26.7598462Z TIMING: entire_frame_compile:21.62559 code_gen:6.88796 inductor_compile:11.78355 backend_compile:19.00507
2024-06-05T10:00:26.7599922Z STATS: call_* op count: 765 | FakeTensorMode.__torch_dispatch__:33867 | FakeTensor.__torch_dispatch__:3499 | ProxyTorchDispatchMode.__torch_dispatch__:9467
2024-06-05T10:00:26.7601257Z Dynamo produced 7 graphs covering 765 ops with 10 graph breaks (2 unique)
2024-06-05T10:00:30.7937872Z 
2024-06-05T10:00:31.0228827Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/squeezenet1_1-b8a52dc0.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/squeezenet1_1-b8a52dc0.pth
2024-06-05T10:00:31.0387761Z 
2024-06-05T10:00:31.0387897Z 
2024-06-05T10:00:31.0523559Z   0% 0.00/4.73M [00:00<?, ?B/s]
2024-06-05T10:00:31.0524063Z 100% 4.73M/4.73M [00:00<00:00, 371MB/s]
2024-06-05T10:00:31.6263648Z 
2024-06-05T10:00:31.6264349Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:00:31.6279661Z cuda eval  squeezenet1_1                      
2024-06-05T10:00:36.8942915Z E0605 10:00:36.893000 140557570925184 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.06571, (ref-fp64): 0.06713 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:00:36.8944461Z pass
2024-06-05T10:00:36.8948618Z TIMING: code_gen:1.80194 inductor_compile:2.774 backend_compile:3.3595 entire_frame_compile:3.75697
2024-06-05T10:00:36.8949958Z STATS: call_* op count: 132 | FakeTensor.__torch_dispatch__:606 | FakeTensorMode.__torch_dispatch__:3829 | ProxyTorchDispatchMode.__torch_dispatch__:815
2024-06-05T10:00:36.8951150Z Dynamo produced 1 graphs covering 132 ops with 0 graph breaks (0 unique)
2024-06-05T10:00:40.4710355Z 
2024-06-05T10:00:41.9221693Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:00:41.9222074Z 
2024-06-05T10:00:42.3829250Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-06-05T10:00:42.3830153Z 
2024-06-05T10:00:43.0339287Z Loading pipeline components...:  50% 3/6 [00:00<00:00,  6.51it/s]
2024-06-05T10:00:43.0339943Z 
2024-06-05T10:00:43.0340406Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.24it/s]
2024-06-05T10:00:43.0341198Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.40it/s]
2024-06-05T10:00:44.2016758Z 
2024-06-05T10:00:44.2017426Z loading model: 0it [00:03, ?it/s]
2024-06-05T10:00:44.2106683Z cuda eval  stable_diffusion_text_encoder      
2024-06-05T10:00:44.2454400Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2024-06-05T10:01:12.3004235Z pass
2024-06-05T10:01:12.3008642Z TIMING: code_gen:5.58574 inductor_compile:11.77132 backend_compile:22.41721 entire_frame_compile:26.34123
2024-06-05T10:01:12.3010703Z STATS: call_* op count: 1647 | FakeTensorMode.__torch_dispatch__:60048 | FakeTensor.__torch_dispatch__:6582 | ProxyTorchDispatchMode.__torch_dispatch__:21058
2024-06-05T10:01:12.3011993Z Dynamo produced 1 graphs covering 1647 ops with 0 graph breaks (0 unique)
2024-06-05T10:01:16.6344809Z 
2024-06-05T10:01:17.7144281Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:01:17.7144646Z 
2024-06-05T10:01:18.1174727Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-06-05T10:01:18.1175189Z 
2024-06-05T10:01:18.7106983Z Loading pipeline components...:  17% 1/6 [00:00<00:02,  2.48it/s]
2024-06-05T10:01:18.7107507Z 
2024-06-05T10:01:18.8110507Z Loading pipeline components...:  33% 2/6 [00:00<00:02,  1.94it/s]
2024-06-05T10:01:18.8111698Z 
2024-06-05T10:01:18.8116405Z Loading pipeline components...:  83% 5/6 [00:01<00:00,  5.88it/s]
2024-06-05T10:01:18.8117528Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.47it/s]
2024-06-05T10:01:20.2148081Z 
2024-06-05T10:01:20.2148841Z loading model: 0it [00:03, ?it/s]
2024-06-05T10:01:20.2325520Z cuda eval  stable_diffusion_unet              
2024-06-05T10:01:20.2329297Z pass_due_to_skip
2024-06-05T10:01:20.4393775Z TIMING:
2024-06-05T10:01:20.4394169Z STATS: call_* op count: 0
2024-06-05T10:01:20.4395416Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-06-05T10:01:23.7921573Z 
2024-06-05T10:01:25.2949130Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:01:25.2949714Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:01:25.3026770Z cuda eval  timm_efficientnet                  
2024-06-05T10:01:44.6799589Z pass
2024-06-05T10:01:44.6824461Z TIMING: code_gen:4.37586 inductor_compile:7.73497 backend_compile:12.9549 entire_frame_compile:16.31633
2024-06-05T10:01:44.6828425Z STATS: call_* op count: 626 | FakeTensor.__torch_dispatch__:3728 | FakeTensorMode.__torch_dispatch__:30386 | ProxyTorchDispatchMode.__torch_dispatch__:8400
2024-06-05T10:01:44.6830903Z Dynamo produced 1 graphs covering 626 ops with 0 graph breaks (0 unique)
2024-06-05T10:01:48.8559405Z 
2024-06-05T10:01:50.9433814Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:01:50.9434706Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:01:50.9527929Z cuda eval  timm_regnet                        
2024-06-05T10:02:13.9441341Z E0605 10:02:13.943000 140536687641216 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00004 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:02:13.9443907Z pass
2024-06-05T10:02:13.9475586Z TIMING: code_gen:3.05008 inductor_compile:6.92219 backend_compile:13.13337 entire_frame_compile:17.82986
2024-06-05T10:02:13.9478769Z STATS: call_* op count: 918 | FakeTensor.__torch_dispatch__:4471 | FakeTensorMode.__torch_dispatch__:36661 | ProxyTorchDispatchMode.__torch_dispatch__:9787
2024-06-05T10:02:13.9481145Z Dynamo produced 1 graphs covering 918 ops with 0 graph breaks (0 unique)
2024-06-05T10:02:18.0840493Z 
2024-06-05T10:02:19.6092565Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:02:19.6093074Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:02:19.6125316Z cuda eval  timm_resnest                       
2024-06-05T10:02:31.0981469Z E0605 10:02:31.097000 140487968154240 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00016, (ref-fp64): 0.00015 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:02:31.0983165Z pass
2024-06-05T10:02:31.1008845Z TIMING: code_gen:3.74085 inductor_compile:5.63286 backend_compile:8.09997 entire_frame_compile:9.0758
2024-06-05T10:02:31.1010221Z STATS: call_* op count: 296 | FakeTensor.__torch_dispatch__:1895 | FakeTensorMode.__torch_dispatch__:14694 | ProxyTorchDispatchMode.__torch_dispatch__:3822
2024-06-05T10:02:31.1011430Z Dynamo produced 1 graphs covering 296 ops with 0 graph breaks (0 unique)
2024-06-05T10:02:34.9669594Z 
2024-06-05T10:02:36.4518426Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:02:36.4519058Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:02:36.4564193Z cuda eval  timm_vision_transformer            
2024-06-05T10:02:49.8126844Z E0605 10:02:49.811000 140151299854976 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00358, (ref-fp64): 0.00492 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:02:49.8128247Z pass
2024-06-05T10:02:49.8146688Z TIMING: code_gen:2.81918 inductor_compile:5.05742 backend_compile:10.15462 entire_frame_compile:11.7061
2024-06-05T10:02:49.8148055Z STATS: call_* op count: 702 | FakeTensor.__torch_dispatch__:2899 | FakeTensorMode.__torch_dispatch__:22097 | ProxyTorchDispatchMode.__torch_dispatch__:7371
2024-06-05T10:02:49.8149270Z Dynamo produced 1 graphs covering 702 ops with 0 graph breaks (0 unique)
2024-06-05T10:02:53.7546608Z 
2024-06-05T10:03:06.0543909Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:03:06.0544419Z loading model: 0it [00:12, ?it/s]
2024-06-05T10:03:06.0678878Z cuda eval  timm_vision_transformer_large      
2024-06-05T10:03:06.0681257Z pass_due_to_skip
2024-06-05T10:03:06.1778517Z TIMING:
2024-06-05T10:03:06.1778959Z STATS: call_* op count: 0
2024-06-05T10:03:06.1779556Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-06-05T10:03:09.5357272Z 
2024-06-05T10:03:11.1363715Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:03:11.1364219Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:03:11.1412791Z cuda eval  timm_vovnet                        
2024-06-05T10:03:27.0549637Z E0605 10:03:27.053000 140650956968576 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.03347, (ref-fp64): 0.03296 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:03:27.0552339Z pass
2024-06-05T10:03:27.0586348Z TIMING: code_gen:3.37174 inductor_compile:5.83708 backend_compile:9.41201 entire_frame_compile:12.2714
2024-06-05T10:03:27.0587701Z STATS: call_* op count: 340 | FakeTensor.__torch_dispatch__:2291 | FakeTensorMode.__torch_dispatch__:19840 | ProxyTorchDispatchMode.__torch_dispatch__:5252
2024-06-05T10:03:27.0588905Z Dynamo produced 1 graphs covering 340 ops with 0 graph breaks (0 unique)
2024-06-05T10:03:31.0421264Z 
2024-06-05T10:03:31.3030968Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:03:31.3031354Z 
2024-06-05T10:03:31.3104820Z clip_merges.bpe: 0.00B [00:00, ?B/s]
2024-06-05T10:03:31.3105310Z clip_merges.bpe: 532kB [00:00, 72.4MB/s]
2024-06-05T10:03:32.8150848Z 
2024-06-05T10:03:32.8151273Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:03:32.8233132Z cuda eval  torch_multimodal_clip              
2024-06-05T10:03:56.4352241Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg301_1)
2024-06-05T10:03:56.7080234Z E0605 10:03:56.707000 140546849301120 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00029 and shape=torch.Size([4, 512]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:03:56.7085411Z E0605 10:03:56.708000 140546849301120 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00040, (ref-fp64): 0.00048 and shape=torch.Size([32, 512]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:03:56.7088472Z pass
2024-06-05T10:03:56.7114169Z TIMING: code_gen:6.67841 inductor_compile:11.73948 backend_compile:19.67182 entire_frame_compile:21.35287
2024-06-05T10:03:56.7115632Z STATS: call_* op count: 72 | FakeTensor.__torch_dispatch__:5948 | FakeTensorMode.__torch_dispatch__:48688 | ProxyTorchDispatchMode.__torch_dispatch__:17964
2024-06-05T10:03:56.7116818Z Dynamo produced 1 graphs covering 72 ops with 0 graph breaks (0 unique)
2024-06-05T10:04:00.9873176Z 
2024-06-05T10:04:01.3373105Z loading model: 0it [00:00, ?it/s]WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-06-05T10:04:01.4912296Z 
2024-06-05T10:04:01.4913254Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:04:01.4914018Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-06-05T10:04:01.4914777Z cuda eval  tts_angular                        
2024-06-05T10:04:01.5547972Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-06-05T10:04:04.5623887Z pass
2024-06-05T10:04:04.5636239Z TIMING: entire_frame_compile:1.71772 inductor_compile:1.41314 backend_compile:1.59052 code_gen:1.24366
2024-06-05T10:04:04.5637673Z STATS: call_* op count: 6 | FakeTensorMode.__torch_dispatch__:425 | ProxyTorchDispatchMode.__torch_dispatch__:54 | FakeTensor.__torch_dispatch__:14
2024-06-05T10:04:04.5638840Z Dynamo produced 3 graphs covering 6 ops with 2 graph breaks (1 unique)
2024-06-05T10:04:08.0014996Z 
2024-06-05T10:04:08.9454991Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2024-06-05T10:04:08.9621855Z 
2024-06-05T10:04:08.9621868Z 
2024-06-05T10:04:09.0626471Z   0% 0.00/528M [00:00<?, ?B/s]
2024-06-05T10:04:09.0626907Z 
2024-06-05T10:04:09.1625864Z   9% 45.8M/528M [00:00<00:01, 479MB/s]
2024-06-05T10:04:09.1626251Z 
2024-06-05T10:04:09.2626540Z  18% 92.5M/528M [00:00<00:00, 485MB/s]
2024-06-05T10:04:09.2627010Z 
2024-06-05T10:04:09.3626907Z  26% 140M/528M [00:00<00:00, 489MB/s] 
2024-06-05T10:04:09.3627920Z 
2024-06-05T10:04:09.4628956Z  35% 186M/528M [00:00<00:00, 490MB/s]
2024-06-05T10:04:09.4629473Z 
2024-06-05T10:04:09.5629617Z  44% 234M/528M [00:00<00:00, 491MB/s]
2024-06-05T10:04:09.5630152Z 
2024-06-05T10:04:09.6633592Z  53% 280M/528M [00:00<00:00, 491MB/s]
2024-06-05T10:04:09.6634105Z 
2024-06-05T10:04:09.7634024Z  62% 327M/528M [00:00<00:00, 491MB/s]
2024-06-05T10:04:09.7634553Z 
2024-06-05T10:04:09.8634379Z  71% 374M/528M [00:00<00:00, 491MB/s]
2024-06-05T10:04:09.8634885Z 
2024-06-05T10:04:09.9636647Z  80% 421M/528M [00:00<00:00, 492MB/s]
2024-06-05T10:04:09.9637153Z 
2024-06-05T10:04:10.0640862Z  89% 468M/528M [00:01<00:00, 492MB/s]
2024-06-05T10:04:10.0641263Z 
2024-06-05T10:04:10.0911526Z  98% 515M/528M [00:01<00:00, 492MB/s]
2024-06-05T10:04:10.0912086Z 100% 528M/528M [00:01<00:00, 490MB/s]
2024-06-05T10:04:10.9333041Z 
2024-06-05T10:04:10.9334402Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:04:10.9339637Z cuda eval  vgg16                              
2024-06-05T10:04:22.7821289Z E0605 10:04:22.781000 139836259570304 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00907, (ref-fp64): 0.00911 and shape=torch.Size([4, 1000]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:04:22.7823241Z pass
2024-06-05T10:04:22.7860584Z TIMING: code_gen:1.50654 inductor_compile:1.95937 backend_compile:3.45728 entire_frame_compile:3.69911
2024-06-05T10:04:22.7862361Z STATS: call_* op count: 80 | FakeTensor.__torch_dispatch__:374 | FakeTensorMode.__torch_dispatch__:2908 | ProxyTorchDispatchMode.__torch_dispatch__:651
2024-06-05T10:04:22.7864063Z Dynamo produced 1 graphs covering 80 ops with 0 graph breaks (0 unique)
2024-06-05T10:04:26.3512352Z 
2024-06-05T10:04:26.9626624Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth
2024-06-05T10:04:26.9791650Z 
2024-06-05T10:04:26.9791656Z 
2024-06-05T10:04:27.0795518Z   0% 0.00/170M [00:00<?, ?B/s]
2024-06-05T10:04:27.0795946Z 
2024-06-05T10:04:27.1798121Z  27% 45.6M/170M [00:00<00:00, 478MB/s]
2024-06-05T10:04:27.1798522Z 
2024-06-05T10:04:27.2800484Z  54% 92.5M/170M [00:00<00:00, 485MB/s]
2024-06-05T10:04:27.2800818Z 
2024-06-05T10:04:27.3455793Z  82% 139M/170M [00:00<00:00, 487MB/s] 
2024-06-05T10:04:27.3456324Z 100% 170M/170M [00:00<00:00, 486MB/s]
2024-06-05T10:04:28.8342080Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-06-05T10:04:29.2364506Z 
2024-06-05T10:04:29.2365114Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:04:29.2365919Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-06-05T10:04:29.2431374Z cuda eval  vision_maskrcnn                    
2024-06-05T10:04:31.3815429Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-06-05T10:04:52.8201708Z W0605 10:04:52.819000 140361796199040 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T10:04:52.8452612Z W0605 10:04:52.844000 140361796199040 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T10:04:52.8700970Z W0605 10:04:52.869000 140361796199040 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T10:04:52.8948212Z W0605 10:04:52.894000 140361796199040 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T10:04:52.9184880Z W0605 10:04:52.918000 140361796199040 torch/_inductor/utils.py:1189] [3/0_1] DeviceCopy in input program
2024-06-05T10:04:54.7210162Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg6_1). Found from : 
2024-06-05T10:04:54.7212830Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2024-06-05T10:04:54.7213892Z     anchors = self.anchor_generator(images, features)
2024-06-05T10:04:54.7215232Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:04:54.7216284Z     return forward_call(*args, **kwargs)
2024-06-05T10:04:54.7217436Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2024-06-05T10:04:54.7218452Z     self.set_cell_anchors(dtype, device)
2024-06-05T10:04:54.7219588Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2024-06-05T10:04:54.7221137Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-06-05T10:04:54.7222833Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2024-06-05T10:04:54.7224272Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-06-05T10:04:54.7224907Z 
2024-06-05T10:04:55.7722911Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0] Graph break from `Tensor.item()`, consider setting:
2024-06-05T10:04:55.7724552Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-06-05T10:04:55.7725682Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0] or:
2024-06-05T10:04:55.7727155Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-06-05T10:04:55.7728538Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0] to include these operations in the captured graph.
2024-06-05T10:04:55.7729655Z W0605 10:04:55.771000 140361796199040 torch/_dynamo/variables/tensor.py:715] [13/0] 
2024-06-05T10:05:40.0452252Z pass
2024-06-05T10:05:40.0453020Z TIMING: entire_frame_compile:66.67565 code_gen:9.32913 inductor_compile:21.78632 backend_compile:36.68791
2024-06-05T10:05:40.0458382Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:87337 | FakeTensor.__torch_dispatch__:7899 | ProxyTorchDispatchMode.__torch_dispatch__:18376 | attempt fast:428 | fast is_contiguous:284 | slow both tensors nontrivially broadcast:144
2024-06-05T10:05:40.0460024Z Dynamo produced 25 graphs covering 1358 ops with 17 graph breaks (2 unique)
2024-06-05T10:05:45.0393977Z 
2024-06-05T10:05:46.9807588Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:05:46.9808123Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:05:46.9895815Z cuda eval  yolov3                             
2024-06-05T10:06:09.2033914Z W0605 10:06:09.202000 139961963127424 torch/_inductor/utils.py:1189] [7/0] DeviceCopy in input program
2024-06-05T10:06:09.3810104Z W0605 10:06:09.380000 139961963127424 torch/_inductor/utils.py:1189] [7/0] DeviceCopy in input program
2024-06-05T10:06:10.8265636Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg1_1). Found from : 
2024-06-05T10:06:10.8267126Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-06-05T10:06:10.8270315Z     self.create_grids((nx, ny), p.device)
2024-06-05T10:06:10.8271344Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-06-05T10:06:10.8272372Z     self.anchor_vec = self.anchor_vec.to(device)
2024-06-05T10:06:10.8272737Z 
2024-06-05T10:06:12.9517677Z W0605 10:06:12.951000 139961963127424 torch/_inductor/utils.py:1189] [7/1] DeviceCopy in input program
2024-06-05T10:06:13.1048349Z W0605 10:06:13.104000 139961963127424 torch/_inductor/utils.py:1189] [7/1] DeviceCopy in input program
2024-06-05T10:06:14.6546389Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg3_1). Found from : 
2024-06-05T10:06:14.6547763Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-06-05T10:06:14.6549081Z     self.create_grids((nx, ny), p.device)
2024-06-05T10:06:14.6550120Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-06-05T10:06:14.6551274Z     self.anchor_vec = self.anchor_vec.to(device)
2024-06-05T10:06:14.6551672Z 
2024-06-05T10:06:53.7882370Z E0605 10:06:53.787000 139961963127424 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01904, (ref-fp64): 0.01998 and shape=torch.Size([4, 12096, 85]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.000100
2024-06-05T10:06:53.7895579Z pass
2024-06-05T10:06:53.8014117Z TIMING: entire_frame_compile:54.19221 code_gen:12.13236 inductor_compile:14.23287 backend_compile:16.86398
2024-06-05T10:06:53.8016062Z STATS: call_* op count: 120 | FakeTensor.__torch_dispatch__:3116 | FakeTensorMode.__torch_dispatch__:18519 | ProxyTorchDispatchMode.__torch_dispatch__:895 | attempt fast:51 | fast is_contiguous:39 | slow no contiguity match:12
2024-06-05T10:06:53.8017712Z Dynamo produced 27 graphs covering 120 ops with 2 graph breaks (1 unique)
2024-06-05T10:06:55.1817977Z accuracy            pass_rate=82.98%
2024-06-05T10:06:55.1821165Z calls_captured      gmean=0.00x mean=725.128x
2024-06-05T10:06:55.1824250Z unique_graphs       gmean=0.00x mean=2.255x
2024-06-05T10:06:55.1827535Z graph_breaks        gmean=0.00x mean=0.766x
2024-06-05T10:06:55.1830666Z unique_graph_breaks gmean=0.00x mean=0.191x
2024-06-05T10:06:55.1834208Z autograd_captures   gmean=0.00x mean=0.000x
2024-06-05T10:06:55.1837415Z autograd_compiles   gmean=0.00x mean=0.000x
2024-06-05T10:06:55.1840920Z cudagraph_skips     gmean=0.00x mean=0.191x
2024-06-05T10:06:56.0510259Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/inductor_torchbench_inference.csv
2024-06-05T10:06:56.3850247Z lennard_jones                       PASS
2024-06-05T10:06:56.3853372Z llama                               PASS
2024-06-05T10:06:56.3858436Z llama_v2_7b_16h                     XFAIL
2024-06-05T10:06:56.3863263Z llava                               XFAIL
2024-06-05T10:06:56.3868160Z maml                                XFAIL
2024-06-05T10:06:56.3873051Z maml_omniglot                       PASS
2024-06-05T10:06:56.3877884Z mnasnet1_0                          PASS
2024-06-05T10:06:56.3882685Z mobilenet_v2                        PASS
2024-06-05T10:06:56.3888297Z mobilenet_v2_quantized_qat          XFAIL
2024-06-05T10:06:56.3892749Z mobilenet_v3_large                  PASS
2024-06-05T10:06:56.3897686Z moco                                PASS
2024-06-05T10:06:56.3902476Z moondream                           XFAIL
2024-06-05T10:06:56.3907269Z nanogpt                             PASS
2024-06-05T10:06:56.3912144Z nvidia_deeprecommender              PASS
2024-06-05T10:06:56.3916735Z opacus_cifar10                      PASS
2024-06-05T10:06:56.3921508Z phlippe_densenet                    PASS
2024-06-05T10:06:56.3926584Z phlippe_resnet                      PASS
2024-06-05T10:06:56.3931800Z pyhpc_equation_of_state             PASS
2024-06-05T10:06:56.3936284Z pyhpc_isoneutral_mixing             PASS
2024-06-05T10:06:56.3941280Z pyhpc_turbulent_kinetic_energy      PASS
2024-06-05T10:06:56.3946030Z pytorch_CycleGAN_and_pix2pix        PASS
2024-06-05T10:06:56.3950762Z pytorch_stargan                     PASS
2024-06-05T10:06:56.3955404Z pytorch_unet                        PASS
2024-06-05T10:06:56.3960202Z resnet152                           PASS
2024-06-05T10:06:56.3965412Z resnet18                            PASS
2024-06-05T10:06:56.3970054Z resnet50                            PASS
2024-06-05T10:06:56.3974826Z resnet50_quantized_qat              XFAIL
2024-06-05T10:06:56.3979727Z resnext50_32x4d                     PASS
2024-06-05T10:06:56.3984526Z sam                                 PASS
2024-06-05T10:06:56.3989073Z sam_fast                            PASS
2024-06-05T10:06:56.3994046Z shufflenet_v2_x1_0                  PASS
2024-06-05T10:06:56.3998982Z soft_actor_critic                   PASS
2024-06-05T10:06:56.4003686Z speech_transformer                  PASS
2024-06-05T10:06:56.4008610Z squeezenet1_1                       PASS
2024-06-05T10:06:56.4013482Z stable_diffusion_text_encoder       PASS
2024-06-05T10:06:56.4018240Z stable_diffusion_unet               XFAIL
2024-06-05T10:06:56.4022928Z timm_efficientnet                   PASS
2024-06-05T10:06:56.4027729Z timm_regnet                         PASS
2024-06-05T10:06:56.4032620Z timm_resnest                        PASS
2024-06-05T10:06:56.4037405Z timm_vision_transformer             PASS
2024-06-05T10:06:56.4042314Z timm_vision_transformer_large       XFAIL
2024-06-05T10:06:56.4047610Z timm_vovnet                         PASS
2024-06-05T10:06:56.4052733Z torch_multimodal_clip               PASS
2024-06-05T10:06:56.4057476Z tts_angular                         PASS
2024-06-05T10:06:56.4062354Z vgg16                               PASS
2024-06-05T10:06:56.4066756Z vision_maskrcnn                     PASS
2024-06-05T10:06:56.4071184Z yolov3                              PASS
2024-06-05T10:06:56.4540990Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/inductor_torchbench_inference.csv
2024-06-05T10:06:56.7957717Z lennard_jones                       PASS
2024-06-05T10:06:56.7961850Z llama                               PASS
2024-06-05T10:06:56.7964787Z llama_v2_7b_16h                     PASS
2024-06-05T10:06:56.7969128Z llava                               PASS
2024-06-05T10:06:56.7973615Z maml                                PASS
2024-06-05T10:06:56.7977948Z maml_omniglot                       PASS
2024-06-05T10:06:56.7982294Z mnasnet1_0                          PASS
2024-06-05T10:06:56.7986519Z mobilenet_v2                        PASS
2024-06-05T10:06:56.7990913Z mobilenet_v2_quantized_qat          PASS
2024-06-05T10:06:56.7995193Z mobilenet_v3_large                  PASS
2024-06-05T10:06:56.7999380Z moco                                PASS
2024-06-05T10:06:56.8004052Z moondream                           PASS
2024-06-05T10:06:56.8008346Z nanogpt                             PASS
2024-06-05T10:06:56.8012730Z nvidia_deeprecommender              PASS
2024-06-05T10:06:56.8016977Z opacus_cifar10                      PASS
2024-06-05T10:06:56.8021309Z phlippe_densenet                    PASS
2024-06-05T10:06:56.8025538Z phlippe_resnet                      PASS
2024-06-05T10:06:56.8029746Z pyhpc_equation_of_state             PASS
2024-06-05T10:06:56.8034479Z pyhpc_isoneutral_mixing             PASS
2024-06-05T10:06:56.8038477Z pyhpc_turbulent_kinetic_energy      PASS
2024-06-05T10:06:56.8043014Z pytorch_CycleGAN_and_pix2pix        PASS
2024-06-05T10:06:56.8047653Z pytorch_stargan                     PASS
2024-06-05T10:06:56.8052038Z pytorch_unet                        PASS
2024-06-05T10:06:56.8056076Z resnet152                           PASS
2024-06-05T10:06:56.8060371Z resnet18                            PASS
2024-06-05T10:06:56.8064645Z resnet50                            PASS
2024-06-05T10:06:56.8068892Z resnet50_quantized_qat              PASS
2024-06-05T10:06:56.8073403Z resnext50_32x4d                     PASS
2024-06-05T10:06:56.8077552Z sam                                 PASS
2024-06-05T10:06:56.8081827Z sam_fast                            PASS
2024-06-05T10:06:56.8086693Z shufflenet_v2_x1_0                  PASS
2024-06-05T10:06:56.8090942Z soft_actor_critic                   PASS
2024-06-05T10:06:56.8095112Z speech_transformer                  PASS
2024-06-05T10:06:56.8099373Z squeezenet1_1                       PASS
2024-06-05T10:06:56.8103621Z stable_diffusion_text_encoder       PASS
2024-06-05T10:06:56.8107849Z stable_diffusion_unet               PASS
2024-06-05T10:06:56.8112121Z timm_efficientnet                   PASS
2024-06-05T10:06:56.8116449Z timm_regnet                         PASS
2024-06-05T10:06:56.8120694Z timm_resnest                        PASS
2024-06-05T10:06:56.8125286Z timm_vision_transformer             PASS
2024-06-05T10:06:56.8129697Z timm_vision_transformer_large       PASS
2024-06-05T10:06:56.8133960Z timm_vovnet                         PASS
2024-06-05T10:06:56.8138205Z torch_multimodal_clip               PASS
2024-06-05T10:06:56.8142475Z tts_angular                         PASS
2024-06-05T10:06:56.8146744Z vgg16                               PASS
2024-06-05T10:06:56.8151143Z vision_maskrcnn                     PASS
2024-06-05T10:06:56.8155292Z yolov3                              PASS
2024-06-05T10:06:56.8628434Z + test_single_dynamo_benchmark training torchbench 1 --training --amp
2024-06-05T10:06:56.8633178Z ++ pwd
2024-06-05T10:06:56.8635335Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-06-05T10:06:56.8636076Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-06-05T10:06:56.8646038Z + local name=training
2024-06-05T10:06:56.8646796Z + shift
2024-06-05T10:06:56.8647184Z + local suite=torchbench
2024-06-05T10:06:56.8647547Z + shift
2024-06-05T10:06:56.8649212Z + local shard_id=1
2024-06-05T10:06:56.8649702Z + shift
2024-06-05T10:06:56.8650039Z + partition_flags=()
2024-06-05T10:06:56.8650414Z + local partition_flags
2024-06-05T10:06:56.8651077Z + [[ -n 2 ]]
2024-06-05T10:06:56.8651557Z + [[ -n 1 ]]
2024-06-05T10:06:56.8652353Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-06-05T10:06:56.8653153Z + [[ inductor_torchbench == *perf_compare* ]]
2024-06-05T10:06:56.8653710Z + [[ inductor_torchbench == *perf* ]]
2024-06-05T10:06:56.8654199Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-06-05T10:06:56.8655839Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv
2024-06-05T10:07:02.1299850Z 
2024-06-05T10:07:02.5679070Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:02.5679621Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:02.5680099Z cuda train lennard_jones                      
2024-06-05T10:07:08.9039945Z W0605 10:07:08.903000 140459548553856 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:07:13.4681549Z pass
2024-06-05T10:07:13.4682471Z TIMING: entire_frame_compile:8.29751 code_gen:4.2006 inductor_compile:5.05253 backend_compile:7.71481
2024-06-05T10:07:13.4683785Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:872 | FakeTensorMode.__torch_dispatch__:5263 | ProxyTorchDispatchMode.__torch_dispatch__:1238
2024-06-05T10:07:13.4684961Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique)
2024-06-05T10:07:17.0620438Z 
2024-06-05T10:07:17.3370822Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:17.3371375Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:17.3372762Z cuda train llava                              
2024-06-05T10:07:17.3375748Z Traceback (most recent call last):
2024-06-05T10:07:17.3376634Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T10:07:17.3377363Z     ) = runner.load_model(
2024-06-05T10:07:17.3378327Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 303, in load_model
2024-06-05T10:07:17.3379255Z     benchmark = benchmark_cls(
2024-06-05T10:07:17.3380280Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-06-05T10:07:17.3381277Z     obj = type.__call__(cls, *args, **kwargs)
2024-06-05T10:07:17.3382205Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__
2024-06-05T10:07:17.3383494Z     super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T10:07:17.3384932Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-06-05T10:07:17.3386527Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-06-05T10:07:17.3387695Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-06-05T10:07:17.3388626Z     self._skip_by_device_name()
2024-06-05T10:07:17.3389557Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-06-05T10:07:17.3390942Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-06-05T10:07:17.3392180Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2024-06-05T10:07:17.3392840Z 
2024-06-05T10:07:17.3392973Z model_fail_to_load
2024-06-05T10:07:20.5107094Z 
2024-06-05T10:07:21.1022702Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:21.1023541Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:21.1024035Z cuda train maml_omniglot                      
2024-06-05T10:07:29.3380528Z W0605 10:07:29.337000 140543795024512 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:07:35.5925026Z E0605 10:07:35.591000 140543795024512 torch/_dynamo/utils.py:1482] key: 4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00019, (ref-fp64): 0.00019 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:07:35.5934398Z E0605 10:07:35.592000 140543795024512 torch/_dynamo/utils.py:1482] key: 8.weight.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00021 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:07:35.5945058Z E0605 10:07:35.594000 140543795024512 torch/_dynamo/utils.py:1482] key: 0.weight, passes_test: True, RMSE (res-fp64): 0.00118, (ref-fp64): 0.00118 and shape=torch.Size([64, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:07:35.5958998Z E0605 10:07:35.595000 140543795024512 torch/_dynamo/utils.py:1482] key: 4.weight, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00077 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:07:35.5969681Z E0605 10:07:35.596000 140543795024512 torch/_dynamo/utils.py:1482] key: 8.weight, passes_test: True, RMSE (res-fp64): 0.00009, (ref-fp64): 0.00014 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:07:35.5994864Z pass
2024-06-05T10:07:35.5995649Z TIMING: entire_frame_compile:9.38023 code_gen:7.88114 inductor_compile:9.17754 backend_compile:8.56687
2024-06-05T10:07:35.5996959Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1255 | FakeTensorMode.__torch_dispatch__:7314 | ProxyTorchDispatchMode.__torch_dispatch__:1538
2024-06-05T10:07:35.5998149Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique)
2024-06-05T10:07:39.3221729Z 
2024-06-05T10:07:41.6676370Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:07:41.6676891Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:07:41.6677361Z cuda train mnasnet1_0                         
2024-06-05T10:08:13.1917122Z W0605 10:08:13.190000 139776683963008 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:08:57.7294465Z E0605 10:08:57.728000 139776683963008 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 18.19447, (ref-fp64): 16.10112 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7298617Z E0605 10:08:57.729000 139776683963008 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.17218, (ref-fp64): 2.63768 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7309203Z E0605 10:08:57.730000 139776683963008 torch/_dynamo/utils.py:1482] key: classifier.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00379, (ref-fp64): 0.00317 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7313077Z E0605 10:08:57.730000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00501, (ref-fp64): 0.01071 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7317221Z E0605 10:08:57.731000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05417, (ref-fp64): 0.04361 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7321404Z E0605 10:08:57.731000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.12430, (ref-fp64): 0.31671 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7326694Z E0605 10:08:57.732000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02678, (ref-fp64): 0.03095 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7331111Z E0605 10:08:57.732000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.06308, (ref-fp64): 0.12865 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7334177Z E0605 10:08:57.732000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.08278, (ref-fp64): 0.11725 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7338500Z E0605 10:08:57.733000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00932, (ref-fp64): 0.01888 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7342367Z E0605 10:08:57.733000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03910, (ref-fp64): 0.07048 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7346507Z E0605 10:08:57.734000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.05757, (ref-fp64): 0.08874 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7350755Z E0605 10:08:57.734000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.02047, (ref-fp64): 0.03285 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7354761Z E0605 10:08:57.735000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04129, (ref-fp64): 0.07691 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7358777Z E0605 10:08:57.735000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.06263, (ref-fp64): 0.09192 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7363260Z E0605 10:08:57.735000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00927, (ref-fp64): 0.01301 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7367729Z E0605 10:08:57.736000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03002, (ref-fp64): 0.05731 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7371687Z E0605 10:08:57.736000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.04469, (ref-fp64): 0.06193 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7376002Z E0605 10:08:57.737000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.01615, (ref-fp64): 0.01863 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7380127Z E0605 10:08:57.737000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.02082, (ref-fp64): 0.05527 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7384010Z E0605 10:08:57.737000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.02771, (ref-fp64): 0.04765 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7388689Z E0605 10:08:57.738000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01224, (ref-fp64): 0.01822 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7392614Z E0605 10:08:57.738000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04026, (ref-fp64): 0.08167 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7396592Z E0605 10:08:57.739000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.07265, (ref-fp64): 0.12304 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7400812Z E0605 10:08:57.739000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01503, (ref-fp64): 0.01719 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7405214Z E0605 10:08:57.740000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.04201, (ref-fp64): 0.07127 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7409322Z E0605 10:08:57.740000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.08786, (ref-fp64): 0.10195 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7413575Z E0605 10:08:57.740000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.01785, (ref-fp64): 0.02330 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7417694Z E0605 10:08:57.741000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.01706, (ref-fp64): 0.04592 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7421925Z E0605 10:08:57.741000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.06136, (ref-fp64): 0.07264 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7426435Z E0605 10:08:57.742000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01876, (ref-fp64): 0.02265 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7430441Z E0605 10:08:57.742000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.03616, (ref-fp64): 0.07163 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7434458Z E0605 10:08:57.743000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.13747, (ref-fp64): 0.14894 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7439112Z E0605 10:08:57.743000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.03072, (ref-fp64): 0.03128 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7443248Z E0605 10:08:57.743000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05526, (ref-fp64): 0.09904 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7447701Z E0605 10:08:57.744000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.11280, (ref-fp64): 0.11506 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7452210Z E0605 10:08:57.744000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.06421, (ref-fp64): 0.06794 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7456243Z E0605 10:08:57.745000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.04139, (ref-fp64): 0.06575 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7460317Z E0605 10:08:57.745000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.07247, (ref-fp64): 0.06909 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7464791Z E0605 10:08:57.746000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03237, (ref-fp64): 0.03779 and shape=torch.Size([96, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7468779Z E0605 10:08:57.746000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.06427, (ref-fp64): 0.10381 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7473003Z E0605 10:08:57.746000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.15616, (ref-fp64): 0.15558 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7477541Z E0605 10:08:57.747000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02918, (ref-fp64): 0.02906 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7481641Z E0605 10:08:57.747000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.06223, (ref-fp64): 0.09528 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7486030Z E0605 10:08:57.748000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.20811, (ref-fp64): 0.22857 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7490370Z E0605 10:08:57.748000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.07688, (ref-fp64): 0.07911 and shape=torch.Size([576, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7494491Z E0605 10:08:57.749000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.02922, (ref-fp64): 0.04700 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7498603Z E0605 10:08:57.749000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.13881, (ref-fp64): 0.14167 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7503052Z E0605 10:08:57.749000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.04088, (ref-fp64): 0.04263 and shape=torch.Size([96, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7507167Z E0605 10:08:57.750000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.05856, (ref-fp64): 0.10501 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7511045Z E0605 10:08:57.750000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.29793, (ref-fp64): 0.30436 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7515854Z E0605 10:08:57.751000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.05961, (ref-fp64): 0.05746 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7519900Z E0605 10:08:57.751000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.07950, (ref-fp64): 0.12337 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7524369Z E0605 10:08:57.751000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.28202, (ref-fp64): 0.28731 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7528492Z E0605 10:08:57.752000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.01927, (ref-fp64): 0.02062 and shape=torch.Size([576, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7532518Z E0605 10:08:57.752000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03426, (ref-fp64): 0.06570 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7536540Z E0605 10:08:57.753000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.09308, (ref-fp64): 0.08836 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7541569Z E0605 10:08:57.753000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03032, (ref-fp64): 0.03566 and shape=torch.Size([192, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7545613Z E0605 10:08:57.754000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04566, (ref-fp64): 0.08846 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7549627Z E0605 10:08:57.754000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.14419, (ref-fp64): 0.12059 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7553975Z E0605 10:08:57.754000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02104, (ref-fp64): 0.01876 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7558283Z E0605 10:08:57.755000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03964, (ref-fp64): 0.05402 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7562592Z E0605 10:08:57.755000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.11232, (ref-fp64): 0.10640 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7566986Z E0605 10:08:57.756000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.02113, (ref-fp64): 0.02065 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7571346Z E0605 10:08:57.756000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.02053, (ref-fp64): 0.04424 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7575282Z E0605 10:08:57.757000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.07519, (ref-fp64): 0.07790 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7579905Z E0605 10:08:57.757000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.02190, (ref-fp64): 0.02270 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7583931Z E0605 10:08:57.757000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.03646, (ref-fp64): 0.08237 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7588257Z E0605 10:08:57.758000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.22919, (ref-fp64): 0.21496 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7592670Z E0605 10:08:57.758000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02416, (ref-fp64): 0.02100 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7596608Z E0605 10:08:57.759000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03174, (ref-fp64): 0.07457 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7600625Z E0605 10:08:57.759000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.09180, (ref-fp64): 0.10296 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7605085Z E0605 10:08:57.760000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.01993, (ref-fp64): 0.02124 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7608887Z E0605 10:08:57.760000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.01564, (ref-fp64): 0.03809 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7613557Z E0605 10:08:57.760000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.06440, (ref-fp64): 0.06634 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7618212Z E0605 10:08:57.761000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01560, (ref-fp64): 0.01653 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7621776Z E0605 10:08:57.761000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.02726, (ref-fp64): 0.06571 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7625815Z E0605 10:08:57.762000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.12929, (ref-fp64): 0.11300 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7630142Z E0605 10:08:57.762000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02447, (ref-fp64): 0.01645 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7634120Z E0605 10:08:57.762000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03557, (ref-fp64): 0.07445 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7638101Z E0605 10:08:57.763000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.10986, (ref-fp64): 0.07294 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7642476Z E0605 10:08:57.763000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.02127, (ref-fp64): 0.01569 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7646952Z E0605 10:08:57.764000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.01665, (ref-fp64): 0.05674 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7650946Z E0605 10:08:57.764000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.07578, (ref-fp64): 0.04985 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7655169Z E0605 10:08:57.765000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01727, (ref-fp64): 0.01732 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7659170Z E0605 10:08:57.765000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.02300, (ref-fp64): 0.06063 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7662988Z E0605 10:08:57.765000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.10615, (ref-fp64): 0.08603 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7667596Z E0605 10:08:57.766000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.03675, (ref-fp64): 0.02948 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7671755Z E0605 10:08:57.766000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03685, (ref-fp64): 0.08828 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7676051Z E0605 10:08:57.767000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.12754, (ref-fp64): 0.12435 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7680379Z E0605 10:08:57.767000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.03667, (ref-fp64): 0.03642 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7684477Z E0605 10:08:57.768000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03205, (ref-fp64): 0.05029 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7688857Z E0605 10:08:57.768000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.10322, (ref-fp64): 0.08071 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7693144Z E0605 10:08:57.768000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01848, (ref-fp64): 0.02309 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7697257Z E0605 10:08:57.769000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.03326, (ref-fp64): 0.04862 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7701301Z E0605 10:08:57.769000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.17148, (ref-fp64): 0.17920 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7705617Z E0605 10:08:57.770000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.14.weight.grad, passes_test: True, RMSE (res-fp64): 0.02222, (ref-fp64): 0.01902 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7709515Z E0605 10:08:57.770000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.15.bias.grad, passes_test: True, RMSE (res-fp64): 0.00375, (ref-fp64): 0.00452 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7713638Z E0605 10:08:57.770000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.15.weight.grad, passes_test: True, RMSE (res-fp64): 0.01431, (ref-fp64): 0.01230 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7717750Z E0605 10:08:57.771000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.06467, (ref-fp64): 0.08048 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7721301Z E0605 10:08:57.771000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.05909, (ref-fp64): 0.06138 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7726024Z E0605 10:08:57.772000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.07541, (ref-fp64): 0.13704 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7730023Z E0605 10:08:57.772000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03783, (ref-fp64): 0.04818 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7733964Z E0605 10:08:57.772000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.06119, (ref-fp64): 0.07795 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7738101Z E0605 10:08:57.773000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.06517, (ref-fp64): 0.14053 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7742370Z E0605 10:08:57.773000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01565, (ref-fp64): 0.02825 and shape=torch.Size([48, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7746324Z E0605 10:08:57.774000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.06146, (ref-fp64): 0.08074 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7750543Z E0605 10:08:57.774000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.07707, (ref-fp64): 0.14639 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7754593Z E0605 10:08:57.775000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.03749, (ref-fp64): 0.05394 and shape=torch.Size([48, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7758837Z E0605 10:08:57.775000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.04280, (ref-fp64): 0.08268 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7763090Z E0605 10:08:57.775000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.07007, (ref-fp64): 0.15082 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7767628Z E0605 10:08:57.776000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03518, (ref-fp64): 0.06100 and shape=torch.Size([24, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7771230Z E0605 10:08:57.776000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04665, (ref-fp64): 0.07933 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7775610Z E0605 10:08:57.777000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.10473, (ref-fp64): 0.17500 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7779780Z E0605 10:08:57.777000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01418, (ref-fp64): 0.02043 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7783914Z E0605 10:08:57.777000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.02390, (ref-fp64): 0.03731 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7788072Z E0605 10:08:57.778000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.07447, (ref-fp64): 0.08450 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7792153Z E0605 10:08:57.778000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.04410, (ref-fp64): 0.05456 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7796145Z E0605 10:08:57.779000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03118, (ref-fp64): 0.04687 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7801247Z E0605 10:08:57.779000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.04411, (ref-fp64): 0.06935 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7805192Z E0605 10:08:57.780000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.02896, (ref-fp64): 0.04418 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7809279Z E0605 10:08:57.780000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04790, (ref-fp64): 0.07669 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7813382Z E0605 10:08:57.780000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.06082, (ref-fp64): 0.11426 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7817410Z E0605 10:08:57.781000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01936, (ref-fp64): 0.02095 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7822092Z E0605 10:08:57.781000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05852, (ref-fp64): 0.06380 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7825891Z E0605 10:08:57.782000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.10631, (ref-fp64): 0.10066 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7830604Z E0605 10:08:57.782000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.06422, (ref-fp64): 0.05017 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7834267Z E0605 10:08:57.782000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.06044, (ref-fp64): 0.07617 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7838536Z E0605 10:08:57.783000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.06396, (ref-fp64): 0.05740 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7842528Z E0605 10:08:57.783000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03516, (ref-fp64): 0.04782 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7847168Z E0605 10:08:57.784000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.06112, (ref-fp64): 0.08759 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7850863Z E0605 10:08:57.784000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.04046, (ref-fp64): 0.06532 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7855063Z E0605 10:08:57.785000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02495, (ref-fp64): 0.03157 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7858822Z E0605 10:08:57.785000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.09915, (ref-fp64): 0.18757 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7862970Z E0605 10:08:57.785000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.13205, (ref-fp64): 0.18512 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7867074Z E0605 10:08:57.786000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.02420, (ref-fp64): 0.03570 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7871044Z E0605 10:08:57.786000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.05697, (ref-fp64): 0.09363 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7875109Z E0605 10:08:57.787000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.09216, (ref-fp64): 0.12425 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7879376Z E0605 10:08:57.787000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03591, (ref-fp64): 0.05614 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7883515Z E0605 10:08:57.787000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.04989, (ref-fp64): 0.08235 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7888068Z E0605 10:08:57.788000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.05962, (ref-fp64): 0.08526 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7892331Z E0605 10:08:57.788000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01214, (ref-fp64): 0.01525 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7896087Z E0605 10:08:57.789000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05512, (ref-fp64): 0.07681 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7900281Z E0605 10:08:57.789000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.07691, (ref-fp64): 0.15105 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7904414Z E0605 10:08:57.789000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.03035, (ref-fp64): 0.04019 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7908290Z E0605 10:08:57.790000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03682, (ref-fp64): 0.05407 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7912377Z E0605 10:08:57.790000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.03049, (ref-fp64): 0.04578 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7916528Z E0605 10:08:57.791000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.01686, (ref-fp64): 0.02595 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7920557Z E0605 10:08:57.791000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.05503, (ref-fp64): 0.08696 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7924555Z E0605 10:08:57.792000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.03309, (ref-fp64): 0.05849 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7929000Z E0605 10:08:57.792000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.01513, (ref-fp64): 0.02000 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7933077Z E0605 10:08:57.792000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05842, (ref-fp64): 0.06700 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7937124Z E0605 10:08:57.793000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.09469, (ref-fp64): 0.08343 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7941289Z E0605 10:08:57.793000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.02916, (ref-fp64): 0.03713 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7945160Z E0605 10:08:57.794000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.03715, (ref-fp64): 0.06823 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7949292Z E0605 10:08:57.794000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.08499, (ref-fp64): 0.10016 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7953554Z E0605 10:08:57.794000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.03383, (ref-fp64): 0.04735 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7957393Z E0605 10:08:57.795000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.7.bias.grad, passes_test: True, RMSE (res-fp64): 0.05521, (ref-fp64): 0.09252 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7961678Z E0605 10:08:57.795000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.7.weight.grad, passes_test: True, RMSE (res-fp64): 0.09809, (ref-fp64): 0.13766 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7972120Z E0605 10:08:57.796000 139776683963008 torch/_dynamo/utils.py:1482] key: classifier.1.weight, passes_test: True, RMSE (res-fp64): 0.00377, (ref-fp64): 0.00390 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.7976271Z E0605 10:08:57.797000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01183, (ref-fp64): 0.01398 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7980855Z E0605 10:08:57.797000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01101, (ref-fp64): 0.01136 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7984965Z E0605 10:08:57.798000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01155, (ref-fp64): 0.01277 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7989821Z E0605 10:08:57.798000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01509, (ref-fp64): 0.01865 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7993687Z E0605 10:08:57.798000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01767, (ref-fp64): 0.01896 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.7998188Z E0605 10:08:57.799000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01728, (ref-fp64): 0.01871 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8002794Z E0605 10:08:57.799000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01618, (ref-fp64): 0.01833 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8007458Z E0605 10:08:57.800000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01690, (ref-fp64): 0.01775 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8011853Z E0605 10:08:57.800000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01719, (ref-fp64): 0.01771 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8016430Z E0605 10:08:57.801000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01644, (ref-fp64): 0.01861 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8020841Z E0605 10:08:57.801000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01758, (ref-fp64): 0.01871 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8025234Z E0605 10:08:57.802000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01596, (ref-fp64): 0.01762 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8029801Z E0605 10:08:57.802000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01269, (ref-fp64): 0.01535 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8034109Z E0605 10:08:57.802000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01593, (ref-fp64): 0.01664 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8038560Z E0605 10:08:57.803000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01551, (ref-fp64): 0.01674 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8043053Z E0605 10:08:57.803000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01418, (ref-fp64): 0.01566 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8047719Z E0605 10:08:57.804000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01729, (ref-fp64): 0.01744 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8052178Z E0605 10:08:57.804000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01644, (ref-fp64): 0.01699 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8056704Z E0605 10:08:57.805000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01362, (ref-fp64): 0.01537 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8060908Z E0605 10:08:57.805000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01750, (ref-fp64): 0.01779 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8065242Z E0605 10:08:57.806000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.1.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01940, (ref-fp64): 0.01740 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8069800Z E0605 10:08:57.806000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01153, (ref-fp64): 0.01425 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8074061Z E0605 10:08:57.806000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01446, (ref-fp64): 0.01538 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8078467Z E0605 10:08:57.807000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01437, (ref-fp64): 0.01495 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8083196Z E0605 10:08:57.807000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01262, (ref-fp64): 0.01439 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8087859Z E0605 10:08:57.808000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01463, (ref-fp64): 0.01570 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8091993Z E0605 10:08:57.808000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01397, (ref-fp64): 0.01491 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8096359Z E0605 10:08:57.809000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01279, (ref-fp64): 0.01439 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8100990Z E0605 10:08:57.809000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01734, (ref-fp64): 0.01848 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8105208Z E0605 10:08:57.810000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.10.2.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01783, (ref-fp64): 0.01799 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8109644Z E0605 10:08:57.810000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01476, (ref-fp64): 0.01789 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8114268Z E0605 10:08:57.810000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01599, (ref-fp64): 0.01887 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8118511Z E0605 10:08:57.811000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01601, (ref-fp64): 0.01883 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8123020Z E0605 10:08:57.811000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01447, (ref-fp64): 0.01755 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8127410Z E0605 10:08:57.812000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01569, (ref-fp64): 0.01789 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8132080Z E0605 10:08:57.812000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01586, (ref-fp64): 0.01800 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8136390Z E0605 10:08:57.813000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01411, (ref-fp64): 0.01691 and shape=torch.Size([96, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8140675Z E0605 10:08:57.813000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01528, (ref-fp64): 0.01720 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8145176Z E0605 10:08:57.814000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01611, (ref-fp64): 0.01741 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8149649Z E0605 10:08:57.814000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01123, (ref-fp64): 0.01373 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8154148Z E0605 10:08:57.814000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01277, (ref-fp64): 0.01473 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8158332Z E0605 10:08:57.815000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01255, (ref-fp64): 0.01478 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8163133Z E0605 10:08:57.815000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01078, (ref-fp64): 0.01382 and shape=torch.Size([576, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8167672Z E0605 10:08:57.816000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01199, (ref-fp64): 0.01483 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8171932Z E0605 10:08:57.816000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01186, (ref-fp64): 0.01401 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8176713Z E0605 10:08:57.817000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01165, (ref-fp64): 0.01418 and shape=torch.Size([96, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8181233Z E0605 10:08:57.817000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01671, (ref-fp64): 0.01881 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8185576Z E0605 10:08:57.818000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.11.1.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01602, (ref-fp64): 0.01761 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8190376Z E0605 10:08:57.818000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01347, (ref-fp64): 0.01706 and shape=torch.Size([576, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8194987Z E0605 10:08:57.819000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01641, (ref-fp64): 0.01899 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8199294Z E0605 10:08:57.819000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01660, (ref-fp64): 0.01921 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8203929Z E0605 10:08:57.819000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01252, (ref-fp64): 0.01524 and shape=torch.Size([576, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8208609Z E0605 10:08:57.820000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01590, (ref-fp64): 0.01831 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8213191Z E0605 10:08:57.820000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01541, (ref-fp64): 0.01736 and shape=torch.Size([576]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8218402Z E0605 10:08:57.821000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01362, (ref-fp64): 0.01609 and shape=torch.Size([192, 576, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8222789Z E0605 10:08:57.821000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01576, (ref-fp64): 0.01764 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8227303Z E0605 10:08:57.822000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01507, (ref-fp64): 0.01679 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8232128Z E0605 10:08:57.822000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01112, (ref-fp64): 0.01303 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8236111Z E0605 10:08:57.823000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01300, (ref-fp64): 0.01469 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8241083Z E0605 10:08:57.823000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01236, (ref-fp64): 0.01439 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8245893Z E0605 10:08:57.824000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01101, (ref-fp64): 0.01317 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8249929Z E0605 10:08:57.824000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01320, (ref-fp64): 0.01565 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8254614Z E0605 10:08:57.825000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01257, (ref-fp64): 0.01477 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8259532Z E0605 10:08:57.825000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01182, (ref-fp64): 0.01417 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8263857Z E0605 10:08:57.825000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01553, (ref-fp64): 0.01818 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8268386Z E0605 10:08:57.826000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.1.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01573, (ref-fp64): 0.01705 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8273125Z E0605 10:08:57.826000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.00846, (ref-fp64): 0.01099 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8277459Z E0605 10:08:57.827000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.00953, (ref-fp64): 0.01205 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8281557Z E0605 10:08:57.827000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.00942, (ref-fp64): 0.01194 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8286924Z E0605 10:08:57.828000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.00836, (ref-fp64): 0.01061 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8290939Z E0605 10:08:57.828000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01007, (ref-fp64): 0.01261 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8295443Z E0605 10:08:57.829000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.00940, (ref-fp64): 0.01187 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8300269Z E0605 10:08:57.829000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.00982, (ref-fp64): 0.01172 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8304385Z E0605 10:08:57.830000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01498, (ref-fp64): 0.01769 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8309124Z E0605 10:08:57.830000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.2.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01524, (ref-fp64): 0.01810 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8313696Z E0605 10:08:57.830000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.00809, (ref-fp64): 0.01068 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8317895Z E0605 10:08:57.831000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.00977, (ref-fp64): 0.01148 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8322574Z E0605 10:08:57.831000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.01153 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8327301Z E0605 10:08:57.832000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.00804, (ref-fp64): 0.01008 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8331714Z E0605 10:08:57.832000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01206 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8336282Z E0605 10:08:57.833000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.00934, (ref-fp64): 0.01108 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8341178Z E0605 10:08:57.833000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.00950, (ref-fp64): 0.01130 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8345513Z E0605 10:08:57.834000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01471, (ref-fp64): 0.01733 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8349959Z E0605 10:08:57.834000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.12.3.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01447, (ref-fp64): 0.01666 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8354838Z E0605 10:08:57.835000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.00886, (ref-fp64): 0.01192 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8359139Z E0605 10:08:57.835000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01232, (ref-fp64): 0.01371 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8363715Z E0605 10:08:57.835000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01226, (ref-fp64): 0.01373 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8368598Z E0605 10:08:57.836000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01023, (ref-fp64): 0.01189 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8372899Z E0605 10:08:57.836000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01249, (ref-fp64): 0.01353 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8377176Z E0605 10:08:57.837000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01151, (ref-fp64): 0.01245 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8382084Z E0605 10:08:57.837000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01252, (ref-fp64): 0.01334 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8386394Z E0605 10:08:57.838000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01840, (ref-fp64): 0.01861 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8390732Z E0605 10:08:57.838000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.13.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01315, (ref-fp64): 0.01518 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8395255Z E0605 10:08:57.839000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.14.weight, passes_test: True, RMSE (res-fp64): 0.00996, (ref-fp64): 0.01316 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8399599Z E0605 10:08:57.839000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.15.bias, passes_test: True, RMSE (res-fp64): 0.01025, (ref-fp64): 0.01252 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8404107Z E0605 10:08:57.839000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.15.weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01235 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8408630Z E0605 10:08:57.840000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01171, (ref-fp64): 0.01216 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8413111Z E0605 10:08:57.840000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01264, (ref-fp64): 0.01323 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8417342Z E0605 10:08:57.841000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01232, (ref-fp64): 0.01405 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8421881Z E0605 10:08:57.841000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01131, (ref-fp64): 0.01377 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8426068Z E0605 10:08:57.842000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01382, (ref-fp64): 0.01806 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8430383Z E0605 10:08:57.842000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01247, (ref-fp64): 0.01719 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8434811Z E0605 10:08:57.843000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01408, (ref-fp64): 0.01753 and shape=torch.Size([48, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8439215Z E0605 10:08:57.843000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.01760 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8443491Z E0605 10:08:57.843000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01502, (ref-fp64): 0.01710 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8448433Z E0605 10:08:57.844000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01511, (ref-fp64): 0.01727 and shape=torch.Size([48, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8452592Z E0605 10:08:57.844000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01524, (ref-fp64): 0.01806 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8456848Z E0605 10:08:57.845000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01595, (ref-fp64): 0.01833 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8461360Z E0605 10:08:57.845000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01627, (ref-fp64): 0.01888 and shape=torch.Size([24, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8465533Z E0605 10:08:57.846000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01777, (ref-fp64): 0.02003 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8469820Z E0605 10:08:57.846000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01650, (ref-fp64): 0.02007 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8474535Z E0605 10:08:57.847000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01352, (ref-fp64): 0.01744 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8478859Z E0605 10:08:57.847000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01578, (ref-fp64): 0.01865 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8483485Z E0605 10:08:57.847000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01508, (ref-fp64): 0.01816 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8488260Z E0605 10:08:57.848000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01429, (ref-fp64): 0.01706 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8492413Z E0605 10:08:57.848000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01581, (ref-fp64): 0.01855 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8496662Z E0605 10:08:57.849000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01511, (ref-fp64): 0.01818 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8501101Z E0605 10:08:57.849000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01580, (ref-fp64): 0.01825 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8505303Z E0605 10:08:57.850000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01806, (ref-fp64): 0.01894 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8509648Z E0605 10:08:57.850000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.1.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01582, (ref-fp64): 0.01869 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8514183Z E0605 10:08:57.850000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01379, (ref-fp64): 0.01734 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8518443Z E0605 10:08:57.851000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01350, (ref-fp64): 0.01762 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8522932Z E0605 10:08:57.851000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01367, (ref-fp64): 0.01765 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8527860Z E0605 10:08:57.852000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01373, (ref-fp64): 0.01694 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8532167Z E0605 10:08:57.852000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01488, (ref-fp64): 0.01765 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8536544Z E0605 10:08:57.853000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01437, (ref-fp64): 0.01845 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8540937Z E0605 10:08:57.853000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01394, (ref-fp64): 0.01882 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8545133Z E0605 10:08:57.854000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01425, (ref-fp64): 0.02010 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8549653Z E0605 10:08:57.854000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.8.2.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01401, (ref-fp64): 0.01566 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8554276Z E0605 10:08:57.855000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01373, (ref-fp64): 0.01726 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8558640Z E0605 10:08:57.855000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01576, (ref-fp64): 0.01673 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8563061Z E0605 10:08:57.855000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01580, (ref-fp64): 0.01740 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8567840Z E0605 10:08:57.856000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01540, (ref-fp64): 0.01665 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8572145Z E0605 10:08:57.856000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01626, (ref-fp64): 0.01688 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8576454Z E0605 10:08:57.857000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01575, (ref-fp64): 0.01622 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8580829Z E0605 10:08:57.857000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01460, (ref-fp64): 0.01701 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8585154Z E0605 10:08:57.858000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01578, (ref-fp64): 0.01735 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8589508Z E0605 10:08:57.858000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.0.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01584, (ref-fp64): 0.01667 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8593879Z E0605 10:08:57.858000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01358, (ref-fp64): 0.01618 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8598345Z E0605 10:08:57.859000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01563, (ref-fp64): 0.01522 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8602656Z E0605 10:08:57.859000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01517, (ref-fp64): 0.01668 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8607383Z E0605 10:08:57.860000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01365, (ref-fp64): 0.01613 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8611780Z E0605 10:08:57.860000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01371, (ref-fp64): 0.01565 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8615894Z E0605 10:08:57.861000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01549, (ref-fp64): 0.01668 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8620309Z E0605 10:08:57.861000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01231, (ref-fp64): 0.01636 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8624598Z E0605 10:08:57.862000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01259, (ref-fp64): 0.01702 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8628919Z E0605 10:08:57.862000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.1.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01288, (ref-fp64): 0.01590 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8633579Z E0605 10:08:57.862000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.0.weight, passes_test: True, RMSE (res-fp64): 0.01351, (ref-fp64): 0.01689 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8638037Z E0605 10:08:57.863000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.1.bias, passes_test: True, RMSE (res-fp64): 0.01399, (ref-fp64): 0.01749 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8642637Z E0605 10:08:57.863000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.1.weight, passes_test: True, RMSE (res-fp64): 0.01407, (ref-fp64): 0.01716 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8647490Z E0605 10:08:57.864000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.3.weight, passes_test: True, RMSE (res-fp64): 0.01344, (ref-fp64): 0.01643 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:08:57.8652011Z E0605 10:08:57.864000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.4.bias, passes_test: True, RMSE (res-fp64): 0.01404, (ref-fp64): 0.01651 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8656410Z E0605 10:08:57.865000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.4.weight, passes_test: True, RMSE (res-fp64): 0.01344, (ref-fp64): 0.01694 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8660926Z E0605 10:08:57.865000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.6.weight, passes_test: True, RMSE (res-fp64): 0.01272, (ref-fp64): 0.01599 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8665427Z E0605 10:08:57.866000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.7.bias, passes_test: True, RMSE (res-fp64): 0.01406, (ref-fp64): 0.01697 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8669773Z E0605 10:08:57.866000 139776683963008 torch/_dynamo/utils.py:1482] key: layers.9.2.layers.7.weight, passes_test: True, RMSE (res-fp64): 0.01459, (ref-fp64): 0.01800 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:08:57.8982470Z pass
2024-06-05T10:08:57.9010964Z TIMING: entire_frame_compile:56.83543 code_gen:25.85178 inductor_compile:40.8858 backend_compile:47.59758
2024-06-05T10:08:57.9012814Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:14794 | FakeTensorMode.__torch_dispatch__:81915 | ProxyTorchDispatchMode.__torch_dispatch__:17678
2024-06-05T10:08:57.9014037Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique)
2024-06-05T10:09:04.2139219Z 
2024-06-05T10:09:06.1384372Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:09:06.1385327Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:09:06.1386291Z cuda train mobilenet_v2                       
2024-06-05T10:09:38.2040073Z E0605 10:09:38.202000 139687396221568 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00565, (ref-fp64): 0.00539 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:09:38.2194913Z E0605 10:09:38.218000 139687396221568 torch/_dynamo/utils.py:1482] key: features.17.conv.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00001, (ref-fp64): 0.00007 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:09:38.2988920Z pass
2024-06-05T10:09:38.3038387Z TIMING: entire_frame_compile:14.4488 code_gen:10.03027 inductor_compile:18.36352 backend_compile:12.59476
2024-06-05T10:09:38.3040519Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:7942 | FakeTensorMode.__torch_dispatch__:42586 | ProxyTorchDispatchMode.__torch_dispatch__:10584
2024-06-05T10:09:38.3041754Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique)
2024-06-05T10:09:42.8367260Z 
2024-06-05T10:09:44.7187483Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:09:44.7188146Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:09:44.7188692Z cuda train mobilenet_v2_quantized_qat         
2024-06-05T10:09:44.7189216Z Traceback (most recent call last):
2024-06-05T10:09:44.7190000Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model
2024-06-05T10:09:44.7191014Z     self.model_iter_fn(model, example_inputs)
2024-06-05T10:09:44.7195649Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in forward_and_backward_pass
2024-06-05T10:09:44.7196826Z     pred = mod(*cloned_inputs)
2024-06-05T10:09:44.7198206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 737, in call_wrapped
2024-06-05T10:09:44.7199422Z     return self._wrapped_call(self, *args, **kwargs)
2024-06-05T10:09:44.7200430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__
2024-06-05T10:09:44.7201504Z     raise e
2024-06-05T10:09:44.7202706Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__
2024-06-05T10:09:44.7203743Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2024-06-05T10:09:44.7204981Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
2024-06-05T10:09:44.7205948Z     return self._call_impl(*args, **kwargs)
2024-06-05T10:09:44.7207205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:09:44.7208331Z     return forward_call(*args, **kwargs)
2024-06-05T10:09:44.7208847Z   File "<eval_with_key>.3", line 207, in forward
2024-06-05T10:09:44.7209646Z     activation_post_process_101 = self.activation_post_process_101(classifier_1);  classifier_1 = None
2024-06-05T10:09:44.7210988Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
2024-06-05T10:09:44.7211932Z     return self._call_impl(*args, **kwargs)
2024-06-05T10:09:44.7212918Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:09:44.7213806Z     return forward_call(*args, **kwargs)
2024-06-05T10:09:44.7214943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward
2024-06-05T10:09:44.7215919Z     return torch.fused_moving_avg_obs_fake_quant(
2024-06-05T10:09:44.7216537Z RuntimeError: expected scalar type Float but found Half
2024-06-05T10:09:44.7216948Z 
2024-06-05T10:09:44.7217266Z The above exception was the direct cause of the following exception:
2024-06-05T10:09:44.7217740Z 
2024-06-05T10:09:44.7217909Z Traceback (most recent call last):
2024-06-05T10:09:44.7218608Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T10:09:44.7219321Z     ) = runner.load_model(
2024-06-05T10:09:44.7220041Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model
2024-06-05T10:09:44.7220867Z     self.validate_model(model, example_inputs)
2024-06-05T10:09:44.7221678Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model
2024-06-05T10:09:44.7222508Z     raise RuntimeError("Eager run failed") from e
2024-06-05T10:09:44.7223022Z RuntimeError: Eager run failed
2024-06-05T10:09:44.7223304Z 
2024-06-05T10:09:44.7223442Z eager_fail_to_run
2024-06-05T10:09:47.9971722Z 
2024-06-05T10:09:50.3010991Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:09:50.3011567Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:09:50.3012059Z cuda train mobilenet_v3_large                 
2024-06-05T10:10:27.0111493Z W0605 10:10:27.010000 140095873970816 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:11:14.9222124Z E0605 10:11:14.921000 140095873970816 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 4.30039, (ref-fp64): 9.87210 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9227946Z E0605 10:11:14.922000 140095873970816 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 1.24315, (ref-fp64): 5.33122 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9233513Z E0605 10:11:14.922000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00048, (ref-fp64): 0.00109 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9241558Z E0605 10:11:14.923000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00946, (ref-fp64): 0.02211 and shape=torch.Size([1280, 960]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9252338Z E0605 10:11:14.924000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00103, (ref-fp64): 0.00213 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9257116Z E0605 10:11:14.925000 140095873970816 torch/_dynamo/utils.py:1482] key: features.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.20394, (ref-fp64): 0.23336 and shape=torch.Size([16, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9261827Z E0605 10:11:14.925000 140095873970816 torch/_dynamo/utils.py:1482] key: features.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.16243, (ref-fp64): 2.40620 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9266185Z E0605 10:11:14.926000 140095873970816 torch/_dynamo/utils.py:1482] key: features.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.33658, (ref-fp64): 1.72475 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9271080Z E0605 10:11:14.926000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 2.33567, (ref-fp64): 2.29416 and shape=torch.Size([16, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9276051Z E0605 10:11:14.927000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.92356, (ref-fp64): 2.10900 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9280751Z E0605 10:11:14.927000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.93181, (ref-fp64): 2.20611 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9285824Z E0605 10:11:14.928000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.23335, (ref-fp64): 2.11907 and shape=torch.Size([16, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9290740Z E0605 10:11:14.928000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.06442, (ref-fp64): 2.13773 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9295628Z E0605 10:11:14.929000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.54980, (ref-fp64): 2.62769 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9300691Z E0605 10:11:14.929000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.34165, (ref-fp64): 1.28487 and shape=torch.Size([184, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9305337Z E0605 10:11:14.930000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.36107, (ref-fp64): 0.33574 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9310184Z E0605 10:11:14.930000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 13.21610, (ref-fp64): 11.24493 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9314964Z E0605 10:11:14.931000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 2.86072, (ref-fp64): 2.83719 and shape=torch.Size([184, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9319745Z E0605 10:11:14.931000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.07363, (ref-fp64): 0.09977 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9324379Z E0605 10:11:14.932000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.87786, (ref-fp64): 3.27738 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9329966Z E0605 10:11:14.932000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.41155, (ref-fp64): 1.34294 and shape=torch.Size([80, 184, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9334254Z E0605 10:11:14.933000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05489, (ref-fp64): 0.04877 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9339024Z E0605 10:11:14.933000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 6.68873, (ref-fp64): 5.42887 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9344246Z E0605 10:11:14.933000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.80932, (ref-fp64): 0.62833 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9348813Z E0605 10:11:14.934000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.09169, (ref-fp64): 0.12198 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9353705Z E0605 10:11:14.934000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.66251, (ref-fp64): 4.52680 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9358613Z E0605 10:11:14.935000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.93060, (ref-fp64): 0.87569 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9363546Z E0605 10:11:14.935000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03214, (ref-fp64): 0.03797 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9368586Z E0605 10:11:14.936000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.42029, (ref-fp64): 0.87142 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9373554Z E0605 10:11:14.936000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.01278, (ref-fp64): 0.00797 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9378903Z E0605 10:11:14.937000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.30078, (ref-fp64): 0.18580 and shape=torch.Size([120, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9383714Z E0605 10:11:14.937000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.01695, (ref-fp64): 0.00984 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9388978Z E0605 10:11:14.938000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.87282, (ref-fp64): 0.49328 and shape=torch.Size([480, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9394372Z E0605 10:11:14.938000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.50527, (ref-fp64): 0.58067 and shape=torch.Size([112, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9399017Z E0605 10:11:14.939000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.05728, (ref-fp64): 0.06565 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9404054Z E0605 10:11:14.939000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.24862, (ref-fp64): 2.47033 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9409788Z E0605 10:11:14.940000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.55780, (ref-fp64): 0.67957 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9414553Z E0605 10:11:14.941000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03845, (ref-fp64): 0.05227 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9419309Z E0605 10:11:14.941000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.89295, (ref-fp64): 1.62047 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9424380Z E0605 10:11:14.942000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.72112, (ref-fp64): 0.82704 and shape=torch.Size([672, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9429081Z E0605 10:11:14.942000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.01582, (ref-fp64): 0.01955 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9434027Z E0605 10:11:14.942000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.80787, (ref-fp64): 0.82007 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9438819Z E0605 10:11:14.943000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00930, (ref-fp64): 0.01158 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9444962Z E0605 10:11:14.944000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.27924, (ref-fp64): 0.34661 and shape=torch.Size([168, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9449899Z E0605 10:11:14.944000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.01179, (ref-fp64): 0.01172 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9455814Z E0605 10:11:14.945000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.50443, (ref-fp64): 0.51067 and shape=torch.Size([672, 168, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9461050Z E0605 10:11:14.945000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.17102, (ref-fp64): 0.29421 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9465950Z E0605 10:11:14.946000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.02268, (ref-fp64): 0.02292 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9470720Z E0605 10:11:14.946000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.94342, (ref-fp64): 1.61763 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9476901Z E0605 10:11:14.947000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.31874, (ref-fp64): 0.65431 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9480952Z E0605 10:11:14.947000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03097, (ref-fp64): 0.04360 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9485829Z E0605 10:11:14.948000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.59014, (ref-fp64): 3.59478 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9491217Z E0605 10:11:14.948000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.31073, (ref-fp64): 2.56344 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9495686Z E0605 10:11:14.949000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.04549, (ref-fp64): 0.08012 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9500298Z E0605 10:11:14.949000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.44143, (ref-fp64): 1.08677 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9504899Z E0605 10:11:14.950000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc1.bias.grad, passes_test: False, RMSE (res-fp64): 0.00266, (ref-fp64): 0.00010 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9510511Z E0605 10:11:14.950000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc1.weight.grad, passes_test: False, RMSE (res-fp64): 0.20659, (ref-fp64): 0.00765 and shape=torch.Size([168, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9515381Z E0605 10:11:14.951000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc2.bias.grad, passes_test: False, RMSE (res-fp64): 0.00298, (ref-fp64): 0.00009 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9521112Z E0605 10:11:14.951000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc2.weight.grad, passes_test: False, RMSE (res-fp64): 0.39667, (ref-fp64): 0.01168 and shape=torch.Size([672, 168, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9526881Z E0605 10:11:14.952000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.24349, (ref-fp64): 1.07130 and shape=torch.Size([160, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9531817Z E0605 10:11:14.952000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.09258, (ref-fp64): 0.41663 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9536614Z E0605 10:11:14.953000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.05234, (ref-fp64): 4.58184 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9541627Z E0605 10:11:14.953000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.16614, (ref-fp64): 0.67620 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9546141Z E0605 10:11:14.954000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.03857, (ref-fp64): 0.14946 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9551449Z E0605 10:11:14.954000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.47642, (ref-fp64): 1.78265 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9556153Z E0605 10:11:14.955000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.20373, (ref-fp64): 0.82777 and shape=torch.Size([960, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9560580Z E0605 10:11:14.955000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.01101, (ref-fp64): 0.03833 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9565611Z E0605 10:11:14.956000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.23159, (ref-fp64): 0.98512 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9570493Z E0605 10:11:14.956000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.01144, (ref-fp64): 0.04342 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9575664Z E0605 10:11:14.957000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.13814, (ref-fp64): 0.53781 and shape=torch.Size([240, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9580494Z E0605 10:11:14.957000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.00830, (ref-fp64): 0.02593 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9585643Z E0605 10:11:14.958000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.12346, (ref-fp64): 0.40117 and shape=torch.Size([960, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9590927Z E0605 10:11:14.958000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.05598, (ref-fp64): 0.13956 and shape=torch.Size([160, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9595622Z E0605 10:11:14.959000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.02436, (ref-fp64): 0.04158 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9600435Z E0605 10:11:14.959000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.45663, (ref-fp64): 1.08278 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9605877Z E0605 10:11:14.960000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.07306, (ref-fp64): 0.16176 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9610901Z E0605 10:11:14.960000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.01088, (ref-fp64): 0.02598 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9615695Z E0605 10:11:14.961000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.17343, (ref-fp64): 0.35344 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9620775Z E0605 10:11:14.961000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.08300, (ref-fp64): 0.15916 and shape=torch.Size([960, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9625588Z E0605 10:11:14.962000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00354, (ref-fp64): 0.00610 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9630404Z E0605 10:11:14.962000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.11681, (ref-fp64): 0.27009 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9635145Z E0605 10:11:14.963000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00393, (ref-fp64): 0.00591 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9640361Z E0605 10:11:14.963000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.07275, (ref-fp64): 0.10918 and shape=torch.Size([240, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9645007Z E0605 10:11:14.964000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.00202, (ref-fp64): 0.00418 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9650502Z E0605 10:11:14.964000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.06042, (ref-fp64): 0.12454 and shape=torch.Size([960, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9655483Z E0605 10:11:14.965000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02768, (ref-fp64): 0.09025 and shape=torch.Size([160, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9660402Z E0605 10:11:14.965000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00388, (ref-fp64): 0.01213 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9664916Z E0605 10:11:14.966000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.09097, (ref-fp64): 0.29140 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9669899Z E0605 10:11:14.966000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.02477, (ref-fp64): 0.06757 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9674430Z E0605 10:11:14.967000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00084, (ref-fp64): 0.00219 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9679368Z E0605 10:11:14.967000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.01374, (ref-fp64): 0.04472 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9684364Z E0605 10:11:14.967000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.19368, (ref-fp64): 1.51371 and shape=torch.Size([64, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9689590Z E0605 10:11:14.968000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 4.17325, (ref-fp64): 5.16628 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9693986Z E0605 10:11:14.968000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 5.56827, (ref-fp64): 7.47448 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9698887Z E0605 10:11:14.969000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.92283, (ref-fp64): 2.67883 and shape=torch.Size([64, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9703444Z E0605 10:11:14.969000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.48500, (ref-fp64): 2.04843 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9708049Z E0605 10:11:14.970000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.05416, (ref-fp64): 2.63675 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9712800Z E0605 10:11:14.970000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.15434, (ref-fp64): 1.47837 and shape=torch.Size([24, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9717417Z E0605 10:11:14.971000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.31673, (ref-fp64): 2.97245 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9722077Z E0605 10:11:14.971000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 3.89286, (ref-fp64): 5.08082 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9727596Z E0605 10:11:14.972000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.80899, (ref-fp64): 0.91711 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9732805Z E0605 10:11:14.972000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.64785, (ref-fp64): 3.05150 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9738171Z E0605 10:11:14.973000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 5.24435, (ref-fp64): 7.22141 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9742914Z E0605 10:11:14.973000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 2.18607, (ref-fp64): 2.64133 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9747197Z E0605 10:11:14.974000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.24768, (ref-fp64): 2.29492 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9752036Z E0605 10:11:14.974000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.12460, (ref-fp64): 2.80803 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9757138Z E0605 10:11:14.975000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.36233, (ref-fp64): 1.42190 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9762206Z E0605 10:11:14.975000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.14775, (ref-fp64): 2.39049 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9768250Z E0605 10:11:14.976000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.05036, (ref-fp64): 2.53818 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9772766Z E0605 10:11:14.976000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.68431, (ref-fp64): 0.85472 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9777032Z E0605 10:11:14.977000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.63015, (ref-fp64): 3.27539 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9781910Z E0605 10:11:14.977000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 5.21117, (ref-fp64): 3.56696 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9786596Z E0605 10:11:14.978000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.81895, (ref-fp64): 0.76544 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9791148Z E0605 10:11:14.978000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.99763, (ref-fp64): 2.83481 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9795517Z E0605 10:11:14.979000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 4.05089, (ref-fp64): 3.25077 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9799812Z E0605 10:11:14.979000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00033, (ref-fp64): 0.00113 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9804433Z E0605 10:11:14.980000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00038, (ref-fp64): 0.00132 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9809308Z E0605 10:11:14.980000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.00099, (ref-fp64): 0.00287 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9813680Z E0605 10:11:14.980000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00261, (ref-fp64): 0.00757 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9818158Z E0605 10:11:14.981000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.34568, (ref-fp64): 1.80892 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9822599Z E0605 10:11:14.981000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.24402, (ref-fp64): 1.77544 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9826841Z E0605 10:11:14.982000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.82795, (ref-fp64): 2.97897 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9831438Z E0605 10:11:14.982000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.54909, (ref-fp64): 1.08083 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9835863Z E0605 10:11:14.983000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.33928, (ref-fp64): 2.16092 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9840369Z E0605 10:11:14.983000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.40509, (ref-fp64): 5.03763 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9845155Z E0605 10:11:14.984000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.58414, (ref-fp64): 0.58177 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9850198Z E0605 10:11:14.984000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.00116, (ref-fp64): 1.09956 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9854035Z E0605 10:11:14.984000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.93702, (ref-fp64): 0.94761 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9858516Z E0605 10:11:14.985000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.30822, (ref-fp64): 0.32425 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9863042Z E0605 10:11:14.985000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.21524, (ref-fp64): 0.22440 and shape=torch.Size([32, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9867481Z E0605 10:11:14.986000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.20483, (ref-fp64): 0.21825 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9871692Z E0605 10:11:14.986000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.31026, (ref-fp64): 0.32954 and shape=torch.Size([120, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9876090Z E0605 10:11:14.987000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.61444, (ref-fp64): 0.60753 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9880311Z E0605 10:11:14.987000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.24526, (ref-fp64): 1.42218 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9884938Z E0605 10:11:14.988000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.32550, (ref-fp64): 2.60941 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9889581Z E0605 10:11:14.988000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.10424, (ref-fp64): 0.74706 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9893929Z E0605 10:11:14.988000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 2.14781, (ref-fp64): 1.47996 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9898184Z E0605 10:11:14.989000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 12.20358, (ref-fp64): 5.83308 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9902639Z E0605 10:11:14.989000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.96507, (ref-fp64): 1.21443 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9906993Z E0605 10:11:14.990000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.64875, (ref-fp64): 1.11081 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9911262Z E0605 10:11:14.990000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.74564, (ref-fp64): 1.48633 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9915533Z E0605 10:11:14.991000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.30692, (ref-fp64): 0.24900 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9920017Z E0605 10:11:14.991000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.65717, (ref-fp64): 0.53445 and shape=torch.Size([32, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9924643Z E0605 10:11:14.992000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.25315, (ref-fp64): 0.17344 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9929502Z E0605 10:11:14.992000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.79929, (ref-fp64): 0.55271 and shape=torch.Size([120, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9933885Z E0605 10:11:14.992000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.82062, (ref-fp64): 3.18436 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9938415Z E0605 10:11:14.993000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.62143, (ref-fp64): 1.26053 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9942840Z E0605 10:11:14.993000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.1.weight.grad, passes_test: True, RMSE (res-fp64): 5.00640, (ref-fp64): 9.47520 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9947736Z E0605 10:11:14.994000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.19372, (ref-fp64): 1.94948 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9952147Z E0605 10:11:14.994000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.57501, (ref-fp64): 0.76553 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9956824Z E0605 10:11:14.995000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.87147, (ref-fp64): 2.43614 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9961550Z E0605 10:11:14.995000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.53514, (ref-fp64): 0.78276 and shape=torch.Size([240, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:14.9966231Z E0605 10:11:14.996000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.31611, (ref-fp64): 0.50084 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9970990Z E0605 10:11:14.996000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 1.77091, (ref-fp64): 2.64617 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9975608Z E0605 10:11:14.997000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.96490, (ref-fp64): 1.35445 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9980163Z E0605 10:11:14.997000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.41121, (ref-fp64): 0.61407 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9984659Z E0605 10:11:14.998000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 3.62163, (ref-fp64): 5.71787 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9989852Z E0605 10:11:14.998000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.29751, (ref-fp64): 2.38348 and shape=torch.Size([200, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9994705Z E0605 10:11:14.999000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.06139, (ref-fp64): 1.77654 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:14.9999864Z E0605 10:11:14.999000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 14.45476, (ref-fp64): 23.13538 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0005084Z E0605 10:11:15.000000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.96874, (ref-fp64): 1.34235 and shape=torch.Size([200, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0009991Z E0605 10:11:15.000000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.19448, (ref-fp64): 0.24124 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0014389Z E0605 10:11:15.001000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 2.18294, (ref-fp64): 3.63046 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0019199Z E0605 10:11:15.001000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.17460, (ref-fp64): 0.87383 and shape=torch.Size([80, 200, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0023811Z E0605 10:11:15.001000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.22791, (ref-fp64): 0.17086 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0028385Z E0605 10:11:15.002000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 5.69567, (ref-fp64): 4.59210 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0033223Z E0605 10:11:15.002000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.72405, (ref-fp64): 1.02266 and shape=torch.Size([184, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0037859Z E0605 10:11:15.003000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.86665, (ref-fp64): 0.71262 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0042580Z E0605 10:11:15.003000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.1.weight.grad, passes_test: True, RMSE (res-fp64): 14.97436, (ref-fp64): 13.69874 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0047736Z E0605 10:11:15.004000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 3.80435, (ref-fp64): 1.91821 and shape=torch.Size([184, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0052393Z E0605 10:11:15.004000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.13246, (ref-fp64): 0.10662 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0057060Z E0605 10:11:15.005000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.1.weight.grad, passes_test: True, RMSE (res-fp64): 3.76378, (ref-fp64): 2.70439 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0061962Z E0605 10:11:15.005000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 1.42338, (ref-fp64): 1.19603 and shape=torch.Size([80, 184, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0066759Z E0605 10:11:15.006000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.09826, (ref-fp64): 0.08149 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0071387Z E0605 10:11:15.006000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.1.weight.grad, passes_test: True, RMSE (res-fp64): 6.85328, (ref-fp64): 5.89479 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0077081Z E0605 10:11:15.007000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.0.bias, passes_test: True, RMSE (res-fp64): 0.00047, (ref-fp64): 0.00050 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0085041Z E0605 10:11:15.008000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.0.weight, passes_test: True, RMSE (res-fp64): 0.00152, (ref-fp64): 0.00191 and shape=torch.Size([1280, 960]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0095647Z E0605 10:11:15.009000 140095873970816 torch/_dynamo/utils.py:1482] key: classifier.3.weight, passes_test: True, RMSE (res-fp64): 0.00127, (ref-fp64): 0.00183 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0100615Z E0605 10:11:15.009000 140095873970816 torch/_dynamo/utils.py:1482] key: features.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00590, (ref-fp64): 0.00598 and shape=torch.Size([16, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0109874Z E0605 10:11:15.010000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00429, (ref-fp64): 0.00446 and shape=torch.Size([16, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0116916Z E0605 10:11:15.011000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00333, (ref-fp64): 0.00110 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0122028Z E0605 10:11:15.011000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00219, (ref-fp64): 0.00178 and shape=torch.Size([16, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0127469Z E0605 10:11:15.012000 140095873970816 torch/_dynamo/utils.py:1482] key: features.1.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00234, (ref-fp64): 0.00182 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0134905Z E0605 10:11:15.013000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00542, (ref-fp64): 0.00495 and shape=torch.Size([184, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0139750Z E0605 10:11:15.013000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00362, (ref-fp64): 0.00302 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0144731Z E0605 10:11:15.014000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00483, (ref-fp64): 0.00436 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0150065Z E0605 10:11:15.014000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00446, (ref-fp64): 0.00407 and shape=torch.Size([184, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0155046Z E0605 10:11:15.015000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00518, (ref-fp64): 0.00487 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0160040Z E0605 10:11:15.015000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00476, (ref-fp64): 0.00370 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0165575Z E0605 10:11:15.016000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00460, (ref-fp64): 0.00440 and shape=torch.Size([80, 184, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0170706Z E0605 10:11:15.016000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00198, (ref-fp64): 0.00137 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0175604Z E0605 10:11:15.017000 140095873970816 torch/_dynamo/utils.py:1482] key: features.10.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00514, (ref-fp64): 0.00424 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0180802Z E0605 10:11:15.017000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00480, (ref-fp64): 0.00449 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0185759Z E0605 10:11:15.018000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00302, (ref-fp64): 0.00326 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0190873Z E0605 10:11:15.018000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00400, (ref-fp64): 0.00361 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0195999Z E0605 10:11:15.019000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00400, (ref-fp64): 0.00361 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0200891Z E0605 10:11:15.019000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00351, (ref-fp64): 0.00336 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0206211Z E0605 10:11:15.020000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00438, (ref-fp64): 0.00405 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0211392Z E0605 10:11:15.020000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00386, (ref-fp64): 0.00312 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0216886Z E0605 10:11:15.021000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00476, (ref-fp64): 0.00418 and shape=torch.Size([120, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0221989Z E0605 10:11:15.021000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00206, (ref-fp64): 0.00197 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0227579Z E0605 10:11:15.022000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00205, (ref-fp64): 0.00207 and shape=torch.Size([480, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0233002Z E0605 10:11:15.022000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00309, (ref-fp64): 0.00318 and shape=torch.Size([112, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0238107Z E0605 10:11:15.023000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00272, (ref-fp64): 0.00287 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0243383Z E0605 10:11:15.023000 140095873970816 torch/_dynamo/utils.py:1482] key: features.11.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00541, (ref-fp64): 0.00486 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0249739Z E0605 10:11:15.024000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00525, (ref-fp64): 0.00563 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0254253Z E0605 10:11:15.025000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00303, (ref-fp64): 0.00302 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0259424Z E0605 10:11:15.025000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00431, (ref-fp64): 0.00483 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0264738Z E0605 10:11:15.026000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00396, (ref-fp64): 0.00434 and shape=torch.Size([672, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0269782Z E0605 10:11:15.026000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00336, (ref-fp64): 0.00364 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0275160Z E0605 10:11:15.027000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00442, (ref-fp64): 0.00494 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0280171Z E0605 10:11:15.027000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00311, (ref-fp64): 0.00359 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0286239Z E0605 10:11:15.028000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00380, (ref-fp64): 0.00408 and shape=torch.Size([168, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0291421Z E0605 10:11:15.028000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00242, (ref-fp64): 0.00329 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0297138Z E0605 10:11:15.029000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00195, (ref-fp64): 0.00258 and shape=torch.Size([672, 168, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0302597Z E0605 10:11:15.029000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00276, (ref-fp64): 0.00285 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0307627Z E0605 10:11:15.030000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00194, (ref-fp64): 0.00215 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0312548Z E0605 10:11:15.030000 140095873970816 torch/_dynamo/utils.py:1482] key: features.12.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00438, (ref-fp64): 0.00457 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0318222Z E0605 10:11:15.031000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00642, (ref-fp64): 0.00435 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0323267Z E0605 10:11:15.031000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00281, (ref-fp64): 0.00126 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0328991Z E0605 10:11:15.032000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00534, (ref-fp64): 0.00313 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0334052Z E0605 10:11:15.032000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00498, (ref-fp64): 0.00273 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0339252Z E0605 10:11:15.033000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00196, (ref-fp64): 0.00101 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0344299Z E0605 10:11:15.034000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00584, (ref-fp64): 0.00314 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0349444Z E0605 10:11:15.034000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00373, (ref-fp64): 0.00239 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0355253Z E0605 10:11:15.035000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00450, (ref-fp64): 0.00332 and shape=torch.Size([168, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0360146Z E0605 10:11:15.035000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00146, (ref-fp64): 0.00137 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0366161Z E0605 10:11:15.036000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00128, (ref-fp64): 0.00150 and shape=torch.Size([672, 168, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0372018Z E0605 10:11:15.036000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00168, (ref-fp64): 0.00183 and shape=torch.Size([160, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0377111Z E0605 10:11:15.037000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00134, (ref-fp64): 0.00213 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0382025Z E0605 10:11:15.037000 140095873970816 torch/_dynamo/utils.py:1482] key: features.13.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00263, (ref-fp64): 0.00341 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0387378Z E0605 10:11:15.038000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00303, (ref-fp64): 0.00373 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0392487Z E0605 10:11:15.038000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00167, (ref-fp64): 0.00250 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0397628Z E0605 10:11:15.039000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00233, (ref-fp64): 0.00312 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0402635Z E0605 10:11:15.039000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00256, (ref-fp64): 0.00329 and shape=torch.Size([960, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0408016Z E0605 10:11:15.040000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00187, (ref-fp64): 0.00252 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0413102Z E0605 10:11:15.040000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00195, (ref-fp64): 0.00296 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0418382Z E0605 10:11:15.041000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00174 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0423906Z E0605 10:11:15.041000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00219, (ref-fp64): 0.00275 and shape=torch.Size([240, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0428950Z E0605 10:11:15.042000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00195, (ref-fp64): 0.00224 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0434468Z E0605 10:11:15.043000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00140, (ref-fp64): 0.00168 and shape=torch.Size([960, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0439662Z E0605 10:11:15.043000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00178, (ref-fp64): 0.00182 and shape=torch.Size([160, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0444763Z E0605 10:11:15.044000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00161 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0449782Z E0605 10:11:15.044000 140095873970816 torch/_dynamo/utils.py:1482] key: features.14.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00389, (ref-fp64): 0.00462 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0455157Z E0605 10:11:15.045000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00600, (ref-fp64): 0.00775 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0460069Z E0605 10:11:15.045000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00371, (ref-fp64): 0.00471 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0465314Z E0605 10:11:15.046000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00590, (ref-fp64): 0.00788 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0470331Z E0605 10:11:15.046000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00485, (ref-fp64): 0.00625 and shape=torch.Size([960, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0475464Z E0605 10:11:15.047000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00326, (ref-fp64): 0.00413 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0480313Z E0605 10:11:15.047000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00584, (ref-fp64): 0.00783 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0485449Z E0605 10:11:15.048000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00285, (ref-fp64): 0.00329 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0491010Z E0605 10:11:15.048000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00402, (ref-fp64): 0.00501 and shape=torch.Size([240, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0495925Z E0605 10:11:15.049000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00046, (ref-fp64): 0.00142 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0501550Z E0605 10:11:15.049000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00103, (ref-fp64): 0.00120 and shape=torch.Size([960, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0506943Z E0605 10:11:15.050000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00127, (ref-fp64): 0.00148 and shape=torch.Size([160, 960, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0511868Z E0605 10:11:15.050000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00177 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0516958Z E0605 10:11:15.051000 140095873970816 torch/_dynamo/utils.py:1482] key: features.15.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00210, (ref-fp64): 0.00483 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0522545Z E0605 10:11:15.051000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.0.weight, passes_test: True, RMSE (res-fp64): 0.00301, (ref-fp64): 0.00464 and shape=torch.Size([960, 160, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0527583Z E0605 10:11:15.052000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.1.bias, passes_test: True, RMSE (res-fp64): 0.00103, (ref-fp64): 0.00227 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0532391Z E0605 10:11:15.052000 140095873970816 torch/_dynamo/utils.py:1482] key: features.16.1.weight, passes_test: True, RMSE (res-fp64): 0.00184, (ref-fp64): 0.00353 and shape=torch.Size([960]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0537503Z E0605 10:11:15.053000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00402, (ref-fp64): 0.00359 and shape=torch.Size([64, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0542224Z E0605 10:11:15.053000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00389, (ref-fp64): 0.00334 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0547116Z E0605 10:11:15.054000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00321, (ref-fp64): 0.00272 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0551912Z E0605 10:11:15.054000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00341, (ref-fp64): 0.00348 and shape=torch.Size([64, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0556935Z E0605 10:11:15.055000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00368, (ref-fp64): 0.00271 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0561550Z E0605 10:11:15.055000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00173, (ref-fp64): 0.00280 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0567054Z E0605 10:11:15.056000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00382, (ref-fp64): 0.00351 and shape=torch.Size([24, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0571770Z E0605 10:11:15.056000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00407, (ref-fp64): 0.00340 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0576576Z E0605 10:11:15.057000 140095873970816 torch/_dynamo/utils.py:1482] key: features.2.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00600, (ref-fp64): 0.00484 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0581727Z E0605 10:11:15.057000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00414, (ref-fp64): 0.00396 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0586506Z E0605 10:11:15.058000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00250, (ref-fp64): 0.00275 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0591331Z E0605 10:11:15.058000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00151, (ref-fp64): 0.00237 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0596055Z E0605 10:11:15.059000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00297, (ref-fp64): 0.00317 and shape=torch.Size([72, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0600820Z E0605 10:11:15.059000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00336, (ref-fp64): 0.00375 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0608261Z E0605 10:11:15.060000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00490, (ref-fp64): 0.00382 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0613465Z E0605 10:11:15.060000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00549, (ref-fp64): 0.00354 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0617869Z E0605 10:11:15.061000 140095873970816 torch/_dynamo/utils.py:1482] key: features.3.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00272, (ref-fp64): 0.00361 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0622802Z E0605 10:11:15.061000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00246, (ref-fp64): 0.00245 and shape=torch.Size([72, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0627591Z E0605 10:11:15.062000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00194, (ref-fp64): 0.00128 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0632587Z E0605 10:11:15.062000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00190, (ref-fp64): 0.00142 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0637261Z E0605 10:11:15.063000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00274, (ref-fp64): 0.00243 and shape=torch.Size([72, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0642034Z E0605 10:11:15.063000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00293, (ref-fp64): 0.00277 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0647298Z E0605 10:11:15.064000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00195, (ref-fp64): 0.00196 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0654488Z E0605 10:11:15.065000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00130, (ref-fp64): 0.00133 and shape=torch.Size([24, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0659223Z E0605 10:11:15.065000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00196, (ref-fp64): 0.00196 and shape=torch.Size([72]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0666445Z E0605 10:11:15.066000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00336, (ref-fp64): 0.00309 and shape=torch.Size([40, 72, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0671045Z E0605 10:11:15.066000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00271, (ref-fp64): 0.00199 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0675861Z E0605 10:11:15.067000 140095873970816 torch/_dynamo/utils.py:1482] key: features.4.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00413, (ref-fp64): 0.00416 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0680787Z E0605 10:11:15.067000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00477, (ref-fp64): 0.00462 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0685857Z E0605 10:11:15.068000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00435, (ref-fp64): 0.00506 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0690739Z E0605 10:11:15.068000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00372, (ref-fp64): 0.00352 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0695699Z E0605 10:11:15.069000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00283, (ref-fp64): 0.00290 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0700399Z E0605 10:11:15.069000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00383, (ref-fp64): 0.00389 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0705242Z E0605 10:11:15.070000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00384, (ref-fp64): 0.00405 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0710207Z E0605 10:11:15.070000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00300, (ref-fp64): 0.00396 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0715137Z E0605 10:11:15.071000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00232, (ref-fp64): 0.00311 and shape=torch.Size([32, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0720048Z E0605 10:11:15.071000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00378, (ref-fp64): 0.00371 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0725233Z E0605 10:11:15.072000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00230, (ref-fp64): 0.00227 and shape=torch.Size([120, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0730357Z E0605 10:11:15.072000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00286, (ref-fp64): 0.00257 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0735042Z E0605 10:11:15.073000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00357, (ref-fp64): 0.00264 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0740072Z E0605 10:11:15.073000 140095873970816 torch/_dynamo/utils.py:1482] key: features.5.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00351, (ref-fp64): 0.00332 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0744860Z E0605 10:11:15.074000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00340, (ref-fp64): 0.00351 and shape=torch.Size([120, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0749638Z E0605 10:11:15.074000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00284, (ref-fp64): 0.00302 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0755478Z E0605 10:11:15.075000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00256, (ref-fp64): 0.00268 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0759507Z E0605 10:11:15.075000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00336, (ref-fp64): 0.00329 and shape=torch.Size([120, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0764659Z E0605 10:11:15.076000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00290, (ref-fp64): 0.00276 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0769417Z E0605 10:11:15.076000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00205, (ref-fp64): 0.00203 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0774455Z E0605 10:11:15.077000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00272, (ref-fp64): 0.00147 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0779688Z E0605 10:11:15.077000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00280, (ref-fp64): 0.00182 and shape=torch.Size([32, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0784623Z E0605 10:11:15.078000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00242, (ref-fp64): 0.00204 and shape=torch.Size([120]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0789950Z E0605 10:11:15.078000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00151, (ref-fp64): 0.00120 and shape=torch.Size([120, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0795164Z E0605 10:11:15.079000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.0.weight, passes_test: True, RMSE (res-fp64): 0.00264, (ref-fp64): 0.00198 and shape=torch.Size([40, 120, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0800084Z E0605 10:11:15.079000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.1.bias, passes_test: True, RMSE (res-fp64): 0.00277, (ref-fp64): 0.00149 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0805439Z E0605 10:11:15.080000 140095873970816 torch/_dynamo/utils.py:1482] key: features.6.block.3.1.weight, passes_test: True, RMSE (res-fp64): 0.00396, (ref-fp64): 0.00174 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0810862Z E0605 10:11:15.080000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00351, (ref-fp64): 0.00353 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0815862Z E0605 10:11:15.081000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00263, (ref-fp64): 0.00246 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0820874Z E0605 10:11:15.081000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00301, (ref-fp64): 0.00330 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0826253Z E0605 10:11:15.082000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00262, (ref-fp64): 0.00211 and shape=torch.Size([240, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0831318Z E0605 10:11:15.082000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00287, (ref-fp64): 0.00177 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0836292Z E0605 10:11:15.083000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00224, (ref-fp64): 0.00235 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0841440Z E0605 10:11:15.083000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00306, (ref-fp64): 0.00204 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0846783Z E0605 10:11:15.084000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00248, (ref-fp64): 0.00099 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0851564Z E0605 10:11:15.084000 140095873970816 torch/_dynamo/utils.py:1482] key: features.7.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00415, (ref-fp64): 0.00297 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0856716Z E0605 10:11:15.085000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00500, (ref-fp64): 0.00465 and shape=torch.Size([200, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0861745Z E0605 10:11:15.085000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00416, (ref-fp64): 0.00442 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0866795Z E0605 10:11:15.086000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00515, (ref-fp64): 0.00381 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0871886Z E0605 10:11:15.086000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00390, (ref-fp64): 0.00339 and shape=torch.Size([200, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0876735Z E0605 10:11:15.087000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00583, (ref-fp64): 0.00551 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0881712Z E0605 10:11:15.087000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00438, (ref-fp64): 0.00322 and shape=torch.Size([200]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0887480Z E0605 10:11:15.088000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00488, (ref-fp64): 0.00424 and shape=torch.Size([80, 200, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0892958Z E0605 10:11:15.088000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00349, (ref-fp64): 0.00170 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0897917Z E0605 10:11:15.089000 140095873970816 torch/_dynamo/utils.py:1482] key: features.8.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00438, (ref-fp64): 0.00337 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0903123Z E0605 10:11:15.089000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.0.weight, passes_test: True, RMSE (res-fp64): 0.00563, (ref-fp64): 0.00520 and shape=torch.Size([184, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0908481Z E0605 10:11:15.090000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.1.bias, passes_test: True, RMSE (res-fp64): 0.00454, (ref-fp64): 0.00391 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0913548Z E0605 10:11:15.090000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.0.1.weight, passes_test: True, RMSE (res-fp64): 0.00609, (ref-fp64): 0.00545 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0918609Z E0605 10:11:15.091000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.0.weight, passes_test: True, RMSE (res-fp64): 0.00500, (ref-fp64): 0.00488 and shape=torch.Size([184, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:11:15.0923692Z E0605 10:11:15.091000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.1.bias, passes_test: True, RMSE (res-fp64): 0.00565, (ref-fp64): 0.00551 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0929331Z E0605 10:11:15.092000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.1.1.weight, passes_test: True, RMSE (res-fp64): 0.00525, (ref-fp64): 0.00397 and shape=torch.Size([184]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0934302Z E0605 10:11:15.093000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.0.weight, passes_test: True, RMSE (res-fp64): 0.00426, (ref-fp64): 0.00387 and shape=torch.Size([80, 184, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0939240Z E0605 10:11:15.093000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.1.bias, passes_test: True, RMSE (res-fp64): 0.00293, (ref-fp64): 0.00165 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.0944244Z E0605 10:11:15.094000 140095873970816 torch/_dynamo/utils.py:1482] key: features.9.block.2.1.weight, passes_test: True, RMSE (res-fp64): 0.00510, (ref-fp64): 0.00344 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:11:15.1251968Z pass
2024-06-05T10:11:15.1289266Z TIMING: entire_frame_compile:62.54513 code_gen:27.15023 inductor_compile:45.86193 backend_compile:52.09264
2024-06-05T10:11:15.1290612Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:17708 | FakeTensorMode.__torch_dispatch__:91577 | ProxyTorchDispatchMode.__torch_dispatch__:19585
2024-06-05T10:11:15.1293737Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique)
2024-06-05T10:11:21.9071682Z 
2024-06-05T10:11:24.0288462Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:11:24.0289430Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:11:24.0289903Z cuda train moco                               
2024-06-05T10:11:29.3761017Z [rank0]:W0605 10:11:29.375000 139810180186752 torch/_logging/_internal.py:1033] [3/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:11:35.1902081Z [rank0]:W0605 10:11:35.189000 139810180186752 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2024-06-05T10:11:35.2093473Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2024-06-05T10:11:35.2097111Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐
2024-06-05T10:11:35.2098735Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2024-06-05T10:11:35.2100345Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤
2024-06-05T10:11:35.2101907Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] │       0 │         161 │               94032128 │
2024-06-05T10:11:35.2103575Z [rank0]:W0605 10:11:35.208000 139810180186752 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘
2024-06-05T10:11:59.0628683Z skipping cudagraphs due to mutated inputs (161 instances)
2024-06-05T10:12:00.2454820Z [rank0]:W0605 10:12:00.245000 139810180186752 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:12:00.2568632Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 
2024-06-05T10:12:00.2570454Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp
2024-06-05T10:12:00.2571644Z     idx_shuffle = torch.randperm(batch_size_all).cuda()
2024-06-05T10:12:00.2572085Z 
2024-06-05T10:12:01.2644603Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0] Graph break from `Tensor.item()`, consider setting:
2024-06-05T10:12:01.2646220Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-06-05T10:12:01.2649023Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0] or:
2024-06-05T10:12:01.2650663Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-06-05T10:12:01.2652074Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0] to include these operations in the captured graph.
2024-06-05T10:12:01.2653251Z [rank0]:W0605 10:12:01.263000 139810180186752 torch/_dynamo/variables/tensor.py:715] [7/0] 
2024-06-05T10:12:30.9848344Z [rank0]:E0605 10:12:30.984000 139810180186752 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01930, (ref-fp64): 0.01934 and shape=torch.Size([4, 32001]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:12:31.1902804Z pass
2024-06-05T10:12:31.2217612Z TIMING: entire_frame_compile:43.01605 inductor_compile:32.43668 backend_compile:35.12696 code_gen:20.43599
2024-06-05T10:12:31.2219668Z STATS: call_* op count: 887 | FakeTensorMode.__torch_dispatch__:85106 | ProxyTorchDispatchMode.__torch_dispatch__:20740 | FakeTensor.__torch_dispatch__:13462
2024-06-05T10:12:31.2221295Z Dynamo produced 10 graphs covering 887 ops with 11 graph breaks (8 unique)
2024-06-05T10:12:36.8107779Z 
2024-06-05T10:12:38.6911514Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2024-06-05T10:12:38.9051762Z num decayed parameter tensors: 50, with 124,354,560 parameters
2024-06-05T10:12:38.9052817Z num non-decayed parameter tensors: 98, with 121,344 parameters
2024-06-05T10:12:38.9057506Z using fused AdamW: True
2024-06-05T10:12:39.2946851Z 
2024-06-05T10:12:39.2947582Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:12:39.2948795Z cuda train nanogpt                            
2024-06-05T10:13:05.1333020Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:13:05.1337946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in torch_dynamo_resume_in_forward_and_backward_pass_at_434
2024-06-05T10:13:05.1339186Z     pred = mod(*cloned_inputs)
2024-06-05T10:13:05.1340254Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:13:05.1341150Z     return forward_call(*args, **kwargs)
2024-06-05T10:13:05.1342063Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward
2024-06-05T10:13:05.1346455Z     tok_emb = self.transformer.wte(idx)  # token embeddings of shape (b, t, n_embd)
2024-06-05T10:13:05.1347056Z 
2024-06-05T10:13:05.1805564Z W0605 10:13:05.179000 139856468173440 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:13:41.6393547Z E0605 10:13:41.638000 139856468173440 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.03315, (ref-fp64): 0.10627 and shape=torch.Size([4, 1, 50304]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6398855Z E0605 10:13:41.639000 139856468173440 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00776, (ref-fp64): 0.05881 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.6412103Z E0605 10:13:41.640000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_attn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6416686Z E0605 10:13:41.641000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_proj.bias.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00035 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.6421154Z E0605 10:13:41.641000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00016, (ref-fp64): 0.00027 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6431488Z E0605 10:13:41.642000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.ln_2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00010, (ref-fp64): 0.00016 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.6448426Z E0605 10:13:41.644000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00005, (ref-fp64): 0.00008 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6463067Z E0605 10:13:41.645000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00007, (ref-fp64): 0.00010 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6530450Z E0605 10:13:41.652000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00006 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6550309Z E0605 10:13:41.654000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.attn.c_attn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00004 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6577285Z E0605 10:13:41.657000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6599121Z E0605 10:13:41.659000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00013 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6637204Z E0605 10:13:41.663000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00014 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6679274Z E0605 10:13:41.667000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00009, (ref-fp64): 0.00018 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6706075Z E0605 10:13:41.670000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6719230Z E0605 10:13:41.671000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00016 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6750049Z E0605 10:13:41.674000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00010 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6762476Z E0605 10:13:41.675000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_attn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00006 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6769780Z E0605 10:13:41.676000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00014 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6792711Z E0605 10:13:41.678000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6807168Z E0605 10:13:41.680000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6819395Z E0605 10:13:41.681000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.attn.c_attn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00006 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6846759Z E0605 10:13:41.684000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6860065Z E0605 10:13:41.685000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00007, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6892943Z E0605 10:13:41.688000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_fc.weight.grad, passes_test: True, RMSE (res-fp64): 0.00004, (ref-fp64): 0.00007 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6907185Z E0605 10:13:41.690000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00013 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6948744Z E0605 10:13:41.694000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.mlp.c_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00014 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.6954421Z E0605 10:13:41.694000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.ln_f.weight.grad, passes_test: True, RMSE (res-fp64): 0.00058, (ref-fp64): 0.00158 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.6959517Z E0605 10:13:41.695000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.wpe.weight.grad, passes_test: True, RMSE (res-fp64): 0.00006, (ref-fp64): 0.00009 and shape=torch.Size([1024, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7127867Z E0605 10:13:41.712000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.wte.weight.grad, passes_test: True, RMSE (res-fp64): 0.00001, (ref-fp64): 0.00001 and shape=torch.Size([50304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7132873Z E0605 10:13:41.712000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00885, (ref-fp64): 0.00904 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7142697Z E0605 10:13:41.713000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00533, (ref-fp64): 0.00621 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7146928Z E0605 10:13:41.714000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00205, (ref-fp64): 0.00379 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7151728Z E0605 10:13:41.714000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00332, (ref-fp64): 0.00476 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7156114Z E0605 10:13:41.715000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00227, (ref-fp64): 0.00385 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7160325Z E0605 10:13:41.715000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00482, (ref-fp64): 0.00625 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7164995Z E0605 10:13:41.716000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00286, (ref-fp64): 0.00398 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7169559Z E0605 10:13:41.716000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00369, (ref-fp64): 0.00517 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7173971Z E0605 10:13:41.716000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00275, (ref-fp64): 0.00443 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7186220Z E0605 10:13:41.718000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00396, (ref-fp64): 0.00511 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7190776Z E0605 10:13:41.718000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00213, (ref-fp64): 0.00420 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7203332Z E0605 10:13:41.719000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.0.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00313, (ref-fp64): 0.00420 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7208338Z E0605 10:13:41.720000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00915, (ref-fp64): 0.00925 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7217991Z E0605 10:13:41.721000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00596, (ref-fp64): 0.00664 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7222269Z E0605 10:13:41.721000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00267, (ref-fp64): 0.00365 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7227011Z E0605 10:13:41.722000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00284, (ref-fp64): 0.00417 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7231481Z E0605 10:13:41.722000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00344, (ref-fp64): 0.00468 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7235624Z E0605 10:13:41.723000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00463, (ref-fp64): 0.00586 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7240022Z E0605 10:13:41.723000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00334, (ref-fp64): 0.00473 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7244298Z E0605 10:13:41.724000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00398, (ref-fp64): 0.00506 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7249156Z E0605 10:13:41.724000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00310, (ref-fp64): 0.00458 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7261584Z E0605 10:13:41.725000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00394, (ref-fp64): 0.00528 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7265858Z E0605 10:13:41.726000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00266, (ref-fp64): 0.00397 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7278713Z E0605 10:13:41.727000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.1.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00295, (ref-fp64): 0.00408 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7282900Z E0605 10:13:41.727000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00885, (ref-fp64): 0.00894 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7293256Z E0605 10:13:41.728000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00858, (ref-fp64): 0.00882 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7297851Z E0605 10:13:41.729000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00199, (ref-fp64): 0.00262 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7302778Z E0605 10:13:41.729000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00261, (ref-fp64): 0.00407 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7307233Z E0605 10:13:41.730000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00354, (ref-fp64): 0.00491 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7311728Z E0605 10:13:41.730000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00443, (ref-fp64): 0.00597 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7316181Z E0605 10:13:41.731000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00435, (ref-fp64): 0.00761 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7320660Z E0605 10:13:41.731000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00532, (ref-fp64): 0.00852 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7325535Z E0605 10:13:41.732000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00414, (ref-fp64): 0.00697 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7338088Z E0605 10:13:41.733000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00448, (ref-fp64): 0.00707 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7342559Z E0605 10:13:41.733000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00103 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7355189Z E0605 10:13:41.735000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.10.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00243, (ref-fp64): 0.00353 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7359620Z E0605 10:13:41.735000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00814, (ref-fp64): 0.00819 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7369967Z E0605 10:13:41.736000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00771, (ref-fp64): 0.00785 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7376545Z E0605 10:13:41.737000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00223, (ref-fp64): 0.00307 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7380965Z E0605 10:13:41.737000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00307, (ref-fp64): 0.00309 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7385108Z E0605 10:13:41.738000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00347, (ref-fp64): 0.00428 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7389649Z E0605 10:13:41.738000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00187, (ref-fp64): 0.00322 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7394554Z E0605 10:13:41.738000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00255, (ref-fp64): 0.00396 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7398299Z E0605 10:13:41.739000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00288, (ref-fp64): 0.00285 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7411145Z E0605 10:13:41.740000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00324, (ref-fp64): 0.00372 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7425469Z E0605 10:13:41.742000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.11.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00169, (ref-fp64): 0.00278 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7429907Z E0605 10:13:41.742000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.01029, (ref-fp64): 0.01067 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7439930Z E0605 10:13:41.743000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00772, (ref-fp64): 0.00816 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7444265Z E0605 10:13:41.744000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00222, (ref-fp64): 0.00358 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7449378Z E0605 10:13:41.744000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00294, (ref-fp64): 0.00439 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7453586Z E0605 10:13:41.744000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00283, (ref-fp64): 0.00459 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7458008Z E0605 10:13:41.745000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00377, (ref-fp64): 0.00481 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7462307Z E0605 10:13:41.745000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00368, (ref-fp64): 0.00522 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7466687Z E0605 10:13:41.746000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00402, (ref-fp64): 0.00557 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7470966Z E0605 10:13:41.746000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00394, (ref-fp64): 0.00544 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7483589Z E0605 10:13:41.747000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00447, (ref-fp64): 0.00573 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7488276Z E0605 10:13:41.748000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00154, (ref-fp64): 0.00275 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7500600Z E0605 10:13:41.749000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.2.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00289, (ref-fp64): 0.00414 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7505099Z E0605 10:13:41.750000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.01143, (ref-fp64): 0.01183 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7515120Z E0605 10:13:41.751000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00889, (ref-fp64): 0.00930 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7519278Z E0605 10:13:41.751000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00213, (ref-fp64): 0.00310 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7524305Z E0605 10:13:41.751000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00295, (ref-fp64): 0.00432 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7528675Z E0605 10:13:41.752000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00425, (ref-fp64): 0.00579 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7533189Z E0605 10:13:41.752000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00423, (ref-fp64): 0.00602 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7537399Z E0605 10:13:41.753000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00337, (ref-fp64): 0.00523 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7541789Z E0605 10:13:41.753000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00439, (ref-fp64): 0.00529 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7546068Z E0605 10:13:41.754000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00311, (ref-fp64): 0.00484 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7558627Z E0605 10:13:41.755000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00377, (ref-fp64): 0.00509 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7563066Z E0605 10:13:41.755000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00168, (ref-fp64): 0.00282 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7575656Z E0605 10:13:41.757000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.3.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00288, (ref-fp64): 0.00415 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7580150Z E0605 10:13:41.757000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.01105, (ref-fp64): 0.01111 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7590161Z E0605 10:13:41.758000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00930, (ref-fp64): 0.00967 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7594661Z E0605 10:13:41.759000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00168, (ref-fp64): 0.00356 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7599137Z E0605 10:13:41.759000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00305, (ref-fp64): 0.00471 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7603586Z E0605 10:13:41.759000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00328, (ref-fp64): 0.00509 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7608246Z E0605 10:13:41.760000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00388, (ref-fp64): 0.00555 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7612614Z E0605 10:13:41.760000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00318, (ref-fp64): 0.00533 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7616858Z E0605 10:13:41.761000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00361, (ref-fp64): 0.00571 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7621195Z E0605 10:13:41.761000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00317, (ref-fp64): 0.00526 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7633901Z E0605 10:13:41.762000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00375, (ref-fp64): 0.00542 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7638118Z E0605 10:13:41.763000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00200, (ref-fp64): 0.00359 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7651471Z E0605 10:13:41.764000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.4.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00294, (ref-fp64): 0.00430 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7655504Z E0605 10:13:41.765000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.01125, (ref-fp64): 0.01149 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7665376Z E0605 10:13:41.766000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00963, (ref-fp64): 0.00991 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7669624Z E0605 10:13:41.766000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00223, (ref-fp64): 0.00385 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7674407Z E0605 10:13:41.767000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00277, (ref-fp64): 0.00472 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7678740Z E0605 10:13:41.767000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00342, (ref-fp64): 0.00474 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7683125Z E0605 10:13:41.767000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00443, (ref-fp64): 0.00572 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7687741Z E0605 10:13:41.768000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00358, (ref-fp64): 0.00519 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7692304Z E0605 10:13:41.768000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00361, (ref-fp64): 0.00633 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7696543Z E0605 10:13:41.769000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00334, (ref-fp64): 0.00485 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7709050Z E0605 10:13:41.770000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00377, (ref-fp64): 0.00557 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7713504Z E0605 10:13:41.770000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00233, (ref-fp64): 0.00376 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7726672Z E0605 10:13:41.772000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.5.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00287, (ref-fp64): 0.00421 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7730820Z E0605 10:13:41.772000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.01016, (ref-fp64): 0.01042 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7740831Z E0605 10:13:41.773000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00937, (ref-fp64): 0.00964 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7745185Z E0605 10:13:41.774000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00235, (ref-fp64): 0.00340 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7749887Z E0605 10:13:41.774000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00280, (ref-fp64): 0.00429 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7754228Z E0605 10:13:41.775000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00401, (ref-fp64): 0.00572 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7758911Z E0605 10:13:41.775000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00452, (ref-fp64): 0.00658 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7763297Z E0605 10:13:41.775000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00282, (ref-fp64): 0.00502 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7767833Z E0605 10:13:41.776000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00407, (ref-fp64): 0.00619 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7772223Z E0605 10:13:41.776000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00414, (ref-fp64): 0.00624 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7784900Z E0605 10:13:41.778000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00450, (ref-fp64): 0.00657 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7789215Z E0605 10:13:41.778000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00170, (ref-fp64): 0.00359 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7802069Z E0605 10:13:41.779000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.6.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00266, (ref-fp64): 0.00441 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7806664Z E0605 10:13:41.780000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00978, (ref-fp64): 0.01001 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7816808Z E0605 10:13:41.781000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00910, (ref-fp64): 0.00937 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7821107Z E0605 10:13:41.781000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00243, (ref-fp64): 0.00375 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7825875Z E0605 10:13:41.782000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00296, (ref-fp64): 0.00449 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7830272Z E0605 10:13:41.782000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00314, (ref-fp64): 0.00510 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7834745Z E0605 10:13:41.783000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00405, (ref-fp64): 0.00637 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7839219Z E0605 10:13:41.783000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00426, (ref-fp64): 0.00676 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7843846Z E0605 10:13:41.783000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00378, (ref-fp64): 0.00668 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7848669Z E0605 10:13:41.784000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00412, (ref-fp64): 0.00688 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7861146Z E0605 10:13:41.785000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00453, (ref-fp64): 0.00703 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7865398Z E0605 10:13:41.786000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00192, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7877873Z E0605 10:13:41.787000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.7.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00276, (ref-fp64): 0.00425 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7882168Z E0605 10:13:41.787000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00969, (ref-fp64): 0.01000 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7892528Z E0605 10:13:41.788000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00918, (ref-fp64): 0.00951 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7896689Z E0605 10:13:41.789000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00191, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7901525Z E0605 10:13:41.789000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00290, (ref-fp64): 0.00429 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7905908Z E0605 10:13:41.790000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00361, (ref-fp64): 0.00493 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7910246Z E0605 10:13:41.790000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00442, (ref-fp64): 0.00557 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7914658Z E0605 10:13:41.791000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00397, (ref-fp64): 0.00563 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7919047Z E0605 10:13:41.791000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00421, (ref-fp64): 0.00601 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7923731Z E0605 10:13:41.791000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00364, (ref-fp64): 0.00662 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7936301Z E0605 10:13:41.793000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00406, (ref-fp64): 0.00697 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7940619Z E0605 10:13:41.793000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00274 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7953141Z E0605 10:13:41.794000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.8.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00272, (ref-fp64): 0.00457 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7957590Z E0605 10:13:41.795000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.attn.c_attn.bias, passes_test: True, RMSE (res-fp64): 0.00924, (ref-fp64): 0.00949 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7967905Z E0605 10:13:41.796000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.attn.c_attn.weight, passes_test: True, RMSE (res-fp64): 0.00886, (ref-fp64): 0.00915 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7972470Z E0605 10:13:41.796000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.attn.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00179, (ref-fp64): 0.00302 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7977323Z E0605 10:13:41.797000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.attn.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00287, (ref-fp64): 0.00438 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.7981659Z E0605 10:13:41.797000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.ln_1.bias, passes_test: True, RMSE (res-fp64): 0.00352, (ref-fp64): 0.00423 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7986122Z E0605 10:13:41.798000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.ln_1.weight, passes_test: True, RMSE (res-fp64): 0.00448, (ref-fp64): 0.00556 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7990359Z E0605 10:13:41.798000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.ln_2.bias, passes_test: True, RMSE (res-fp64): 0.00383, (ref-fp64): 0.00722 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7994574Z E0605 10:13:41.799000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.ln_2.weight, passes_test: True, RMSE (res-fp64): 0.00476, (ref-fp64): 0.00726 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.7999058Z E0605 10:13:41.799000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.mlp.c_fc.bias, passes_test: True, RMSE (res-fp64): 0.00410, (ref-fp64): 0.00738 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.8012126Z E0605 10:13:41.800000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.mlp.c_fc.weight, passes_test: True, RMSE (res-fp64): 0.00446, (ref-fp64): 0.00772 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.8016311Z E0605 10:13:41.801000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.mlp.c_proj.bias, passes_test: True, RMSE (res-fp64): 0.00149, (ref-fp64): 0.00331 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.8028817Z E0605 10:13:41.802000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.h.9.mlp.c_proj.weight, passes_test: True, RMSE (res-fp64): 0.00287, (ref-fp64): 0.00458 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.8034990Z E0605 10:13:41.803000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.ln_f.weight, passes_test: True, RMSE (res-fp64): 0.00226, (ref-fp64): 0.00294 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:13:41.8040571Z E0605 10:13:41.803000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.wpe.weight, passes_test: True, RMSE (res-fp64): 0.00120, (ref-fp64): 0.00145 and shape=torch.Size([1024, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.8206599Z E0605 10:13:41.820000 139856468173440 torch/_dynamo/utils.py:1482] key: transformer.wte.weight, passes_test: True, RMSE (res-fp64): 0.00079, (ref-fp64): 0.00158 and shape=torch.Size([50304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:13:41.8223393Z pass
2024-06-05T10:13:41.8877583Z TIMING: entire_frame_compile:49.7382 code_gen:19.46857 inductor_compile:34.14676 backend_compile:42.24648
2024-06-05T10:13:41.8878994Z STATS: call_* op count: 859 | FakeTensorMode.__torch_dispatch__:78823 | FakeTensor.__torch_dispatch__:13417 | ProxyTorchDispatchMode.__torch_dispatch__:19295
2024-06-05T10:13:41.8880195Z Dynamo produced 3 graphs covering 859 ops with 7 graph breaks (5 unique)
2024-06-05T10:13:47.9281402Z 
2024-06-05T10:13:50.6031904Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:13:50.6032424Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:13:50.6032918Z cuda train nvidia_deeprecommender             
2024-06-05T10:13:59.2927546Z W0605 10:13:59.291000 140459526148736 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:14:04.4380381Z E0605 10:14:04.437000 140459526148736 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 8.70142, (ref-fp64): 9.81963 and shape=torch.Size([4, 197951]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.4385965Z E0605 10:14:04.438000 140459526148736 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.03254, (ref-fp64): 0.84514 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.4389928Z E0605 10:14:04.438000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_b.0.grad, passes_test: True, RMSE (res-fp64): 0.00007, (ref-fp64): 0.00005 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.4398379Z E0605 10:14:04.439000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.0.grad, passes_test: True, RMSE (res-fp64): 0.05003, (ref-fp64): 0.03623 and shape=torch.Size([512, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.4403058Z E0605 10:14:04.439000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.1.grad, passes_test: True, RMSE (res-fp64): 0.04563, (ref-fp64): 0.07534 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.4836372Z E0605 10:14:04.482000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.2.grad, passes_test: True, RMSE (res-fp64): 0.00038, (ref-fp64): 0.00043 and shape=torch.Size([197951, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.4840303Z E0605 10:14:04.483000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_b.1.grad, passes_test: True, RMSE (res-fp64): 0.00014, (ref-fp64): 0.00015 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.5271546Z E0605 10:14:04.526000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.0.grad, passes_test: True, RMSE (res-fp64): 0.00011, (ref-fp64): 0.00014 and shape=torch.Size([512, 197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5276004Z E0605 10:14:04.527000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.1.grad, passes_test: True, RMSE (res-fp64): 0.08289, (ref-fp64): 0.08926 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5280874Z E0605 10:14:04.527000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.2.grad, passes_test: True, RMSE (res-fp64): 0.03445, (ref-fp64): 0.03230 and shape=torch.Size([1024, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5285478Z E0605 10:14:04.528000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_b.0, passes_test: True, RMSE (res-fp64): 0.00244, (ref-fp64): 0.00262 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.5290356Z E0605 10:14:04.528000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_b.1, passes_test: True, RMSE (res-fp64): 0.00082, (ref-fp64): 0.00064 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.5295195Z E0605 10:14:04.529000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_b.2, passes_test: True, RMSE (res-fp64): 0.00015, (ref-fp64): 0.00017 and shape=torch.Size([197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5299869Z E0605 10:14:04.529000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.0, passes_test: True, RMSE (res-fp64): 0.00282, (ref-fp64): 0.00334 and shape=torch.Size([512, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5304617Z E0605 10:14:04.530000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.1, passes_test: True, RMSE (res-fp64): 0.00302, (ref-fp64): 0.00339 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5733814Z E0605 10:14:04.572000 140459526148736 torch/_dynamo/utils.py:1482] key: decode_w.2, passes_test: True, RMSE (res-fp64): 0.00139, (ref-fp64): 0.00155 and shape=torch.Size([197951, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.5738533Z E0605 10:14:04.573000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_b.0, passes_test: True, RMSE (res-fp64): 0.00198, (ref-fp64): 0.00253 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.5742736Z E0605 10:14:04.573000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_b.1, passes_test: True, RMSE (res-fp64): 0.00202, (ref-fp64): 0.00242 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:04.5747277Z E0605 10:14:04.574000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_b.2, passes_test: True, RMSE (res-fp64): 0.00180, (ref-fp64): 0.00185 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.6177297Z E0605 10:14:04.617000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.0, passes_test: True, RMSE (res-fp64): 0.00239, (ref-fp64): 0.00299 and shape=torch.Size([512, 197951]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.6181667Z E0605 10:14:04.617000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.1, passes_test: True, RMSE (res-fp64): 0.00309, (ref-fp64): 0.00339 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.6186749Z E0605 10:14:04.618000 140459526148736 torch/_dynamo/utils.py:1482] key: encode_w.2, passes_test: True, RMSE (res-fp64): 0.00323, (ref-fp64): 0.00326 and shape=torch.Size([1024, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:04.6190021Z pass
2024-06-05T10:14:04.6457604Z TIMING: entire_frame_compile:10.26817 code_gen:4.47708 inductor_compile:5.65776 backend_compile:9.50015
2024-06-05T10:14:04.6458980Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:6505 | FakeTensor.__torch_dispatch__:1179 | ProxyTorchDispatchMode.__torch_dispatch__:1497
2024-06-05T10:14:04.6460176Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique)
2024-06-05T10:14:08.3012164Z 
2024-06-05T10:14:09.2342929Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:09.2343457Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:09.2343937Z cuda train opacus_cifar10                     
2024-06-05T10:14:09.2354953Z Traceback (most recent call last):
2024-06-05T10:14:09.2355992Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model
2024-06-05T10:14:09.2356873Z     self.model_iter_fn(model, example_inputs)
2024-06-05T10:14:09.2357852Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 441, in forward_and_backward_pass
2024-06-05T10:14:09.2358981Z     self.grad_scaler.scale(loss).backward()
2024-06-05T10:14:09.2360217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 520, in backward
2024-06-05T10:14:09.2361033Z     torch.autograd.backward(
2024-06-05T10:14:09.2361927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward
2024-06-05T10:14:09.2362901Z     _engine_run_backward(
2024-06-05T10:14:09.2363857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward
2024-06-05T10:14:09.2365437Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-06-05T10:14:09.2366986Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 72, in __call__
2024-06-05T10:14:09.2367862Z     return self.hook(module, *args, **kwargs)
2024-06-05T10:14:09.2368995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
2024-06-05T10:14:09.2370081Z     activations, backprops = self.rearrange_grad_samples(
2024-06-05T10:14:09.2371263Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
2024-06-05T10:14:09.2372240Z     raise ValueError(
2024-06-05T10:14:09.2373335Z ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)
2024-06-05T10:14:09.2374094Z 
2024-06-05T10:14:09.2374432Z The above exception was the direct cause of the following exception:
2024-06-05T10:14:09.2374914Z 
2024-06-05T10:14:09.2375075Z Traceback (most recent call last):
2024-06-05T10:14:09.2375777Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T10:14:09.2376479Z     ) = runner.load_model(
2024-06-05T10:14:09.2377190Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model
2024-06-05T10:14:09.2378006Z     self.validate_model(model, example_inputs)
2024-06-05T10:14:09.2378817Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model
2024-06-05T10:14:09.2379632Z     raise RuntimeError("Eager run failed") from e
2024-06-05T10:14:09.2380141Z RuntimeError: Eager run failed
2024-06-05T10:14:09.2380421Z 
2024-06-05T10:14:09.2380552Z eager_fail_to_run
2024-06-05T10:14:12.4955439Z 
2024-06-05T10:14:13.2816738Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:13.2817252Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:13.2817810Z cuda train phlippe_densenet                   
2024-06-05T10:14:41.7925279Z pass
2024-06-05T10:14:41.7929780Z TIMING: entire_frame_compile:16.43334 code_gen:9.96585 inductor_compile:15.99889 backend_compile:14.3343
2024-06-05T10:14:41.7931161Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:6771 | FakeTensorMode.__torch_dispatch__:41752 | ProxyTorchDispatchMode.__torch_dispatch__:11455
2024-06-05T10:14:41.7932373Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique)
2024-06-05T10:14:46.2506163Z 
2024-06-05T10:14:46.9033293Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:46.9033964Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:14:46.9035995Z cuda train phlippe_resnet                     
2024-06-05T10:14:58.6439717Z E0605 10:14:58.643000 139743069971072 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00263, (ref-fp64): 0.00101 and shape=torch.Size([4, 10]). res.dtype: torch.float16, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6447569Z E0605 10:14:58.644000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00061, (ref-fp64): 0.00046 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6450806Z E0605 10:14:58.644000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00144, (ref-fp64): 0.00102 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6455243Z E0605 10:14:58.645000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00098, (ref-fp64): 0.00049 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6459277Z E0605 10:14:58.645000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00052, (ref-fp64): 0.00038 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6463406Z E0605 10:14:58.645000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00056 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6467447Z E0605 10:14:58.646000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.0.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00056 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6471831Z E0605 10:14:58.646000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00051, (ref-fp64): 0.00043 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6475710Z E0605 10:14:58.647000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00050, (ref-fp64): 0.00047 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6479700Z E0605 10:14:58.647000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00066, (ref-fp64): 0.00063 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6483785Z E0605 10:14:58.647000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00081, (ref-fp64): 0.00044 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6488169Z E0605 10:14:58.648000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00079, (ref-fp64): 0.00034 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6492160Z E0605 10:14:58.648000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.1.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00082, (ref-fp64): 0.00045 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6496493Z E0605 10:14:58.649000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00042 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6499848Z E0605 10:14:58.649000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00035 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6504246Z E0605 10:14:58.650000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00081, (ref-fp64): 0.00047 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6508307Z E0605 10:14:58.650000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00078, (ref-fp64): 0.00053 and shape=torch.Size([16, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6512468Z E0605 10:14:58.650000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00069, (ref-fp64): 0.00049 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6516285Z E0605 10:14:58.651000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.2.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00117, (ref-fp64): 0.00048 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6520248Z E0605 10:14:58.651000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.downsample.bias.grad, passes_test: True, RMSE (res-fp64): 0.00035, (ref-fp64): 0.00036 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6524701Z E0605 10:14:58.652000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.downsample.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00049 and shape=torch.Size([32, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6528934Z E0605 10:14:58.652000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00057, (ref-fp64): 0.00042 and shape=torch.Size([32, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6533146Z E0605 10:14:58.652000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00028 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6536673Z E0605 10:14:58.653000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00054, (ref-fp64): 0.00038 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6540980Z E0605 10:14:58.653000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00030 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6544978Z E0605 10:14:58.654000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00035, (ref-fp64): 0.00035 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6549001Z E0605 10:14:58.654000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.3.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00044, (ref-fp64): 0.00031 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6553162Z E0605 10:14:58.654000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.4.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00035, (ref-fp64): 0.00031 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6559210Z E0605 10:14:58.655000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.4.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00030, (ref-fp64): 0.00034 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6563513Z E0605 10:14:58.655000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.4.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00034, (ref-fp64): 0.00025 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6572105Z E0605 10:14:58.656000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.5.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00038, (ref-fp64): 0.00022 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6578059Z E0605 10:14:58.657000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.5.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00034, (ref-fp64): 0.00021 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6582311Z E0605 10:14:58.657000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.5.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00028, (ref-fp64): 0.00021 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6588553Z E0605 10:14:58.658000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.5.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00049, (ref-fp64): 0.00021 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6594772Z E0605 10:14:58.659000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.6.downsample.weight.grad, passes_test: True, RMSE (res-fp64): 0.00034, (ref-fp64): 0.00023 and shape=torch.Size([64, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6598995Z E0605 10:14:58.659000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.6.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00032, (ref-fp64): 0.00022 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6605369Z E0605 10:14:58.660000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.6.net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00033, (ref-fp64): 0.00021 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6609708Z E0605 10:14:58.660000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.6.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00015 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6615704Z E0605 10:14:58.661000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.6.net.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00014 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6619937Z E0605 10:14:58.661000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.7.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00015, (ref-fp64): 0.00011 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6628082Z E0605 10:14:58.662000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.7.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00012, (ref-fp64): 0.00010 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6636045Z E0605 10:14:58.663000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.8.net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00011, (ref-fp64): 0.00010 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6644255Z E0605 10:14:58.663000 139743069971072 torch/_dynamo/utils.py:1482] key: blocks.8.net.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00009, (ref-fp64): 0.00007 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:14:58.6652474Z E0605 10:14:58.664000 139743069971072 torch/_dynamo/utils.py:1482] key: input_net.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00182, (ref-fp64): 0.00155 and shape=torch.Size([16, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6656059Z E0605 10:14:58.665000 139743069971072 torch/_dynamo/utils.py:1482] key: input_net.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00151, (ref-fp64): 0.00092 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6660293Z E0605 10:14:58.665000 139743069971072 torch/_dynamo/utils.py:1482] key: input_net.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00058 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6666226Z E0605 10:14:58.666000 139743069971072 torch/_dynamo/utils.py:1482] key: output_net.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00045, (ref-fp64): 0.00021 and shape=torch.Size([10, 64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:14:58.6909009Z pass
2024-06-05T10:14:58.6909943Z TIMING: entire_frame_compile:6.96071 code_gen:3.14873 inductor_compile:5.40474 backend_compile:6.19604
2024-06-05T10:14:58.6911345Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:2555 | FakeTensorMode.__torch_dispatch__:15865 | ProxyTorchDispatchMode.__torch_dispatch__:4304
2024-06-05T10:14:58.6912630Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique)
2024-06-05T10:15:02.5069948Z 
2024-06-05T10:15:03.7455405Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:15:03.7455970Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:15:03.7456493Z cuda train pytorch_CycleGAN_and_pix2pix       
2024-06-05T10:15:39.5819833Z E0605 10:15:39.581000 139914022130304 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00845, (ref-fp64): 0.01057 and shape=torch.Size([1, 3, 256, 256]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:15:39.5827602Z E0605 10:15:39.582000 139914022130304 torch/_dynamo/utils.py:1482] key: model.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00019, (ref-fp64): 0.00022 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:15:39.5830665Z E0605 10:15:39.582000 139914022130304 torch/_dynamo/utils.py:1482] key: model.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00070, (ref-fp64): 0.00079 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:15:39.5904394Z E0605 10:15:39.589000 139914022130304 torch/_dynamo/utils.py:1482] key: model.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00016, (ref-fp64): 0.00019 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:15:39.5992409Z pass
2024-06-05T10:15:39.6079279Z TIMING: entire_frame_compile:9.14832 code_gen:11.25159 inductor_compile:15.4139 backend_compile:8.44334
2024-06-05T10:15:39.6080753Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:17797 | FakeTensor.__torch_dispatch__:3110 | ProxyTorchDispatchMode.__torch_dispatch__:5123
2024-06-05T10:15:39.6082145Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique)
2024-06-05T10:15:43.7703171Z 
2024-06-05T10:15:45.2286178Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:15:45.2287033Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:15:45.2287633Z cuda train pytorch_stargan                    
2024-06-05T10:16:21.2771607Z W0605 10:16:21.276000 140047957227264 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:16:21.9185693Z W0605 10:16:21.917000 140047957227264 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:16:21.9599792Z W0605 10:16:21.959000 140047957227264 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:16:22.5481946Z W0605 10:16:22.547000 140047957227264 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:16:22.7671919Z CRITICAL:concurrent.futures:Future 140045282436544 in unexpected state: FINISHED
2024-06-05T10:16:22.7711080Z ERROR:common:
2024-06-05T10:16:22.7723228Z Traceback (most recent call last):
2024-06-05T10:16:22.7724039Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2583, in check_accuracy
2024-06-05T10:16:22.7724952Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2024-06-05T10:16:22.7726138Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 421, in _fn
2024-06-05T10:16:22.7810485Z     return fn(*args, **kwargs)
2024-06-05T10:16:22.7811711Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2319, in run_n_iterations
2024-06-05T10:16:22.7812934Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-06-05T10:16:22.7814318Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 433, in forward_and_backward_pass
2024-06-05T10:16:22.7815928Z     cloned_inputs = clone_inputs(inputs)
2024-06-05T10:16:22.7817266Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in torch_dynamo_resume_in_forward_and_backward_pass_at_433
2024-06-05T10:16:22.7818657Z     self.optimizer_zero_grad(mod)
2024-06-05T10:16:22.7820037Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 440, in torch_dynamo_resume_in_forward_and_backward_pass_at_434
2024-06-05T10:16:22.7821436Z     loss = self.compute_loss(pred)
2024-06-05T10:16:22.7869520Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 441, in torch_dynamo_resume_in_forward_and_backward_pass_at_440
2024-06-05T10:16:22.7870990Z     self.grad_scaler.scale(loss).backward()
2024-06-05T10:16:22.7872340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 520, in backward
2024-06-05T10:16:22.7873479Z     torch.autograd.backward(
2024-06-05T10:16:22.7874694Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward
2024-06-05T10:16:22.7875539Z     _engine_run_backward(
2024-06-05T10:16:22.7876487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward
2024-06-05T10:16:22.7877714Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-06-05T10:16:22.7878943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 305, in apply
2024-06-05T10:16:22.7879783Z     return user_fn(self, *args)
2024-06-05T10:16:22.7880844Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1858, in backward
2024-06-05T10:16:22.7881888Z     out = call_compiled_backward()
2024-06-05T10:16:22.7883218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1802, in call_compiled_backward
2024-06-05T10:16:22.7884386Z     CompiledFunction.compiled_bw = aot_config.bw_compiler(
2024-06-05T10:16:22.7885544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 43, in _wrapped_bw_compiler
2024-06-05T10:16:22.7886832Z     return disable(disable(bw_compiler)(*args, **kwargs))
2024-06-05T10:16:22.7888030Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
2024-06-05T10:16:22.7889141Z     return fn(*args, **kwargs)
2024-06-05T10:16:22.7890279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 83, in wrapper_function
2024-06-05T10:16:22.7891316Z     return StrobelightCompileTimeProfiler.profile_compile_time(
2024-06-05T10:16:22.7892588Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time
2024-06-05T10:16:22.7893581Z     return func(*args, **kwargs)
2024-06-05T10:16:22.7894555Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 232, in time_wrapper
2024-06-05T10:16:22.7895475Z     r = func(*args, **kwargs)
2024-06-05T10:16:22.7896418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1423, in bw_compiler
2024-06-05T10:16:22.7897297Z     return inner_compile(
2024-06-05T10:16:22.7898230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
2024-06-05T10:16:22.7899205Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2024-06-05T10:16:22.7900176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 304, in inner
2024-06-05T10:16:22.7900986Z     return fn(*args, **kwargs)
2024-06-05T10:16:22.7901630Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2024-06-05T10:16:22.7902308Z     return func(*args, **kwds)
2024-06-05T10:16:22.7903268Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2024-06-05T10:16:22.7903924Z     return func(*args, **kwds)
2024-06-05T10:16:22.7904817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 232, in time_wrapper
2024-06-05T10:16:22.7905652Z     r = func(*args, **kwargs)
2024-06-05T10:16:22.7906602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 509, in compile_fx_inner
2024-06-05T10:16:22.7907531Z     compiled_graph = fx_codegen_and_compile(
2024-06-05T10:16:22.7908235Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2024-06-05T10:16:22.7909022Z     return func(*args, **kwds)
2024-06-05T10:16:22.7910330Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 805, in fx_codegen_and_compile
2024-06-05T10:16:22.7911770Z     compiled_fn = graph.compile_to_fn()
2024-06-05T10:16:22.7913176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1763, in compile_to_fn
2024-06-05T10:16:22.7914445Z     return self.compile_to_module().call
2024-06-05T10:16:22.7915725Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 232, in time_wrapper
2024-06-05T10:16:22.7916855Z     r = func(*args, **kwargs)
2024-06-05T10:16:22.7918131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1713, in compile_to_module
2024-06-05T10:16:22.7919397Z     mod = PyCodeCache.load_by_key_path(
2024-06-05T10:16:22.7920804Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2862, in load_by_key_path
2024-06-05T10:16:22.7922409Z     mod = _reload_python_module(key, path)
2024-06-05T10:16:22.7924135Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/runtime/compile_tasks.py", line 44, in _reload_python_module
2024-06-05T10:16:22.7925684Z     exec(code, mod.__dict__, mod.__dict__)
2024-06-05T10:16:22.7927377Z   File "/tmp/tmp8sxmbqhi/lh/clh4wvv7ajis6tv6k5tba4nt62wr4wljmfeq5mgifpvvqtknh5ab.py", line 863, in <module>
2024-06-05T10:16:22.7929025Z     triton_poi_fused__to_copy_17 = async_compile.triton('triton_', '''
2024-06-05T10:16:22.7930628Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/async_compile.py", line 157, in triton
2024-06-05T10:16:22.7932023Z     self.process_pool().submit(
2024-06-05T10:16:22.7933502Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 113, in submit
2024-06-05T10:16:22.7934989Z     future.set_running_or_notify_cancel()
2024-06-05T10:16:22.7936405Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 537, in set_running_or_notify_cancel
2024-06-05T10:16:22.7937951Z     raise RuntimeError('Future in unexpected state')
2024-06-05T10:16:22.7938758Z RuntimeError: Future in unexpected state
2024-06-05T10:16:22.7939761Z TorchDynamo optimized model failed to run because of following error
2024-06-05T10:16:22.7940691Z fail_to_run
2024-06-05T10:16:22.7941701Z TIMING: entire_frame_compile:7.83239 code_gen:1.92154 inductor_compile:3.70824 backend_compile:7.11523
2024-06-05T10:16:22.7943595Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:19589 | FakeTensor.__torch_dispatch__:2343 | ProxyTorchDispatchMode.__torch_dispatch__:5427
2024-06-05T10:16:22.7945488Z Dynamo produced 2 graphs covering 60 ops with 5 graph breaks (4 unique)
2024-06-05T10:16:26.7428199Z 
2024-06-05T10:16:27.6100097Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:16:27.6100674Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:16:27.6101163Z cuda train pytorch_unet                       
2024-06-05T10:17:19.0264234Z W0605 10:17:19.025000 140671617950336 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:17:40.9746854Z pass_due_to_skip
2024-06-05T10:17:40.9827591Z TIMING: entire_frame_compile:30.35814 code_gen:19.93808 inductor_compile:28.12011 backend_compile:26.12314
2024-06-05T10:17:40.9828973Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:7166 | FakeTensorMode.__torch_dispatch__:39824 | ProxyTorchDispatchMode.__torch_dispatch__:9107
2024-06-05T10:17:40.9830181Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique)
2024-06-05T10:17:45.8603200Z 
2024-06-05T10:17:48.1597540Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:17:48.1598841Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:17:48.1599441Z cuda train resnet152                          
2024-06-05T10:18:58.4255131Z W0605 10:18:58.424000 140047412257408 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:21:04.8696945Z E0605 10:21:04.868000 140047412257408 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 10328671597717497416986647755950653440.00000, (ref-fp64): 10328671597717497416986647755950653440.00000 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8700324Z E0605 10:21:04.869000 140047412257408 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 9743941719078621999548186634041163776.00000, (ref-fp64): 9743941719078621999548186634041163776.00000 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8705829Z E0605 10:21:04.870000 140047412257408 torch/_dynamo/utils.py:1482] key: bn1.bias.grad, passes_test: True, RMSE (res-fp64): 199979232645733563279643013338365952.00000, (ref-fp64): 199979232645733563279643013338365952.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8709503Z E0605 10:21:04.870000 140047412257408 torch/_dynamo/utils.py:1482] key: bn1.weight.grad, passes_test: True, RMSE (res-fp64): 568417274847048224411161864446148608.00000, (ref-fp64): 568417274847048224411161864446148608.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8714126Z E0605 10:21:04.870000 140047412257408 torch/_dynamo/utils.py:1482] key: conv1.weight.grad, passes_test: True, RMSE (res-fp64): 5415966509405956756603454921637888.00000, (ref-fp64): 5415966509405956756603454921637888.00000 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8729665Z E0605 10:21:04.872000 140047412257408 torch/_dynamo/utils.py:1482] key: fc.weight.grad, passes_test: True, RMSE (res-fp64): 1044139720766333760023717528731648.00000, (ref-fp64): 1044139720766333760023717528731648.00000 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8733817Z E0605 10:21:04.872000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 166431222050800709645744284175958016.00000, (ref-fp64): 166431222050800709645744284175958016.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8737524Z E0605 10:21:04.873000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 230807783489513280289212651999854592.00000, (ref-fp64): 230807783489513280289212651999854592.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8741831Z E0605 10:21:04.873000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 166416098675865820164253784096112640.00000, (ref-fp64): 166416098675865820164253784096112640.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8746046Z E0605 10:21:04.874000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 287183673113477744747461774164361216.00000, (ref-fp64): 287183673113477744747461774164361216.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8750008Z E0605 10:21:04.874000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 91570321305939002474604421282004992.00000, (ref-fp64): 91570321305939002474604421282004992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8754528Z E0605 10:21:04.875000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 211948255280286472513428424180629504.00000, (ref-fp64): 211948255280286472513428424180629504.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8758897Z E0605 10:21:04.875000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 71804966642133336651798393296781312.00000, (ref-fp64): 71804966642133336651798393296781312.00000 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8763795Z E0605 10:21:04.875000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 30008632037502504733309491384156160.00000, (ref-fp64): 30008632037502504733309491384156160.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8768350Z E0605 10:21:04.876000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 71702198327976919772934597104369664.00000, (ref-fp64): 71702198327976919772934597104369664.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8772847Z E0605 10:21:04.876000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 88867857359890790931777595444822016.00000, (ref-fp64): 88867857359890790931777595444822016.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8776822Z E0605 10:21:04.877000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 91570321305939002474604421282004992.00000, (ref-fp64): 91570321305939002474604421282004992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8781007Z E0605 10:21:04.877000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 239512392471229536935003303257833472.00000, (ref-fp64): 239512392471229536935003303257833472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8784805Z E0605 10:21:04.878000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 118738845949867884790043810432286720.00000, (ref-fp64): 118738845949867884790043810432286720.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8789151Z E0605 10:21:04.878000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 212746654828475252086584269634273280.00000, (ref-fp64): 212746654828475252086584269634273280.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8793044Z E0605 10:21:04.878000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 98175779997112958651462161834967040.00000, (ref-fp64): 98175779997112958651462161834967040.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8797249Z E0605 10:21:04.879000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 149508223167153399441604555646697472.00000, (ref-fp64): 149508223167153399441604555646697472.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8800994Z E0605 10:21:04.879000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 108119457002681049990296650649698304.00000, (ref-fp64): 108119457002681049990296650649698304.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8805803Z E0605 10:21:04.880000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 171072842714012028675485437339369472.00000, (ref-fp64): 171072842714012028675485437339369472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8810452Z E0605 10:21:04.880000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 66287207322882679971950068972388352.00000, (ref-fp64): 66287207322882679971950068972388352.00000 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8814114Z E0605 10:21:04.880000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 42650582738633273063939053358940160.00000, (ref-fp64): 42650582738633273063939053358940160.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8818611Z E0605 10:21:04.881000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 41791801179119990936874225529520128.00000, (ref-fp64): 41791801179119990936874225529520128.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8822490Z E0605 10:21:04.881000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 243113709873953707490076874550280192.00000, (ref-fp64): 243113709873953707490076874550280192.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8826637Z E0605 10:21:04.882000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 648159237022147541393100816187916288.00000, (ref-fp64): 648159237022147541393100816187916288.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8830651Z E0605 10:21:04.882000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 232429160799025194942555452403089408.00000, (ref-fp64): 232429160799025194942555452403089408.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8834682Z E0605 10:21:04.883000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1142844146347974609636362202615119872.00000, (ref-fp64): 1142844146347974609636362202615119872.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8838790Z E0605 10:21:04.883000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 99826884577307123218498021977227264.00000, (ref-fp64): 99826884577307123218498021977227264.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8843007Z E0605 10:21:04.883000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 576869269788768410320904062526029824.00000, (ref-fp64): 576869269788768410320904062526029824.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8847612Z E0605 10:21:04.884000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 138181855948457015485157136972906496.00000, (ref-fp64): 138181855948457015485157136972906496.00000 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8851871Z E0605 10:21:04.884000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 129419551354975033171123711133089792.00000, (ref-fp64): 129419551354975033171123711133089792.00000 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8856235Z E0605 10:21:04.885000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 195220794797470077854971502039924736.00000, (ref-fp64): 195220794797470077854971502039924736.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8859873Z E0605 10:21:04.885000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 143434853314860858556208890972733440.00000, (ref-fp64): 143434853314860858556208890972733440.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8864154Z E0605 10:21:04.885000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 593056065642460648488324894931550208.00000, (ref-fp64): 593056065642460648488324894931550208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8868067Z E0605 10:21:04.886000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 63744749370389567468955485327065088.00000, (ref-fp64): 63744749370389567468955485327065088.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8872298Z E0605 10:21:04.886000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 410311544695875268663181948796010496.00000, (ref-fp64): 410311544695875268663181948796010496.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8876151Z E0605 10:21:04.887000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 26849360398294471850923321100075008.00000, (ref-fp64): 26849360398294471850923321100075008.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8880450Z E0605 10:21:04.887000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 354401958036146304595788754447237120.00000, (ref-fp64): 354401958036146304595788754447237120.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8884952Z E0605 10:21:04.888000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 108629961766402135464779066621558784.00000, (ref-fp64): 108629961766402135464779066621558784.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8889527Z E0605 10:21:04.888000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 33984577910942279256350526856495104.00000, (ref-fp64): 33984577910942279256350526856495104.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8894204Z E0605 10:21:04.888000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 58372419930671224757437683644497920.00000, (ref-fp64): 58372419930671224757437683644497920.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8899830Z E0605 10:21:04.889000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 20089776908931327738101322559258624.00000, (ref-fp64): 20089776908931327738101322559258624.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8903789Z E0605 10:21:04.889000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 26849360398294471850923321100075008.00000, (ref-fp64): 26849360398294471850923321100075008.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8908070Z E0605 10:21:04.890000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 128113838418425307664980548392058880.00000, (ref-fp64): 128113838418425307664980548392058880.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8911893Z E0605 10:21:04.890000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 72750883374261367504285931973640192.00000, (ref-fp64): 72750883374261367504285931973640192.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8916204Z E0605 10:21:04.891000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 693225112862392286316127804641509376.00000, (ref-fp64): 693225112862392286316127804641509376.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8919965Z E0605 10:21:04.891000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 43640014055397363358129315869884416.00000, (ref-fp64): 43640014055397363358129315869884416.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8924479Z E0605 10:21:04.891000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 642469918274282082845181959519338496.00000, (ref-fp64): 642469918274282082845181959519338496.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8928629Z E0605 10:21:04.892000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 23674962440417063433274149641912320.00000, (ref-fp64): 23674962440417063433274149641912320.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8933066Z E0605 10:21:04.892000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 729812822615077659727179712114458624.00000, (ref-fp64): 729812822615077659727179712114458624.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8937823Z E0605 10:21:04.893000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 80361741851523820786556121464700928.00000, (ref-fp64): 80361741851523820786556121464700928.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8942352Z E0605 10:21:04.893000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 47587727273796281791837723160477696.00000, (ref-fp64): 47587727273796281791837723160477696.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8947280Z E0605 10:21:04.894000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 98603035965782541184593969973035008.00000, (ref-fp64): 98603035965782541184593969973035008.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8950867Z E0605 10:21:04.894000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 44176255347730535768407754317758464.00000, (ref-fp64): 44176255347730535768407754317758464.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8955310Z E0605 10:21:04.895000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 520548354279337512600093942463070208.00000, (ref-fp64): 520548354279337512600093942463070208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8959077Z E0605 10:21:04.895000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 35734048059292696401440138646781952.00000, (ref-fp64): 35734048059292696401440138646781952.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8963399Z E0605 10:21:04.895000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 562953406214050219725514334783143936.00000, (ref-fp64): 562953406214050219725514334783143936.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8967585Z E0605 10:21:04.896000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 20762589189306056737097834026762240.00000, (ref-fp64): 20762589189306056737097834026762240.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8971561Z E0605 10:21:04.896000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 303543821438829370631330767277391872.00000, (ref-fp64): 303543821438829370631330767277391872.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8976366Z E0605 10:21:04.897000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 77467095047764073160917587482116096.00000, (ref-fp64): 77467095047764073160917587482116096.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8980321Z E0605 10:21:04.897000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 86303965935226554652225272104878080.00000, (ref-fp64): 86303965935226554652225272104878080.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.8985374Z E0605 10:21:04.898000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 82483766334927980305464469596667904.00000, (ref-fp64): 82483766334927980305464469596667904.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8989090Z E0605 10:21:04.898000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 47831174232307535744755609221726208.00000, (ref-fp64): 47831174232307535744755609221726208.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8993430Z E0605 10:21:04.898000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 578584102211152601279112855095869440.00000, (ref-fp64): 578584102211152601279112855095869440.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.8997317Z E0605 10:21:04.899000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 18739686102097563225988255264538624.00000, (ref-fp64): 18739686102097563225988255264538624.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9001585Z E0605 10:21:04.899000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 655937233707216336958752782339276800.00000, (ref-fp64): 655937233707216336958752782339276800.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9005598Z E0605 10:21:04.900000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 14424647034969385216772934486458368.00000, (ref-fp64): 14424647034969385216772934486458368.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9010135Z E0605 10:21:04.900000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 484197296090114737211419094524362752.00000, (ref-fp64): 484197296090114737211419094524362752.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9014932Z E0605 10:21:04.901000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 88219180088227379886362758442123264.00000, (ref-fp64): 88219180088227379886362758442123264.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9019315Z E0605 10:21:04.901000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 39235333780290468261399666999951360.00000, (ref-fp64): 39235333780290468261399666999951360.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9023998Z E0605 10:21:04.901000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 88067258513560509174838450634883072.00000, (ref-fp64): 88067258513560509174838450634883072.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9027846Z E0605 10:21:04.902000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 35780900377027845701617023918800896.00000, (ref-fp64): 35780900377027845701617023918800896.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9032116Z E0605 10:21:04.902000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 853203111397990029397691659622285312.00000, (ref-fp64): 853203111397990029397691659622285312.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9036083Z E0605 10:21:04.903000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 18305958910815595820482074266042368.00000, (ref-fp64): 18305958910815595820482074266042368.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9040167Z E0605 10:21:04.903000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 927187523049219741281994472326955008.00000, (ref-fp64): 927187523049219741281994472326955008.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9044253Z E0605 10:21:04.903000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 7283191157045600716183116389023744.00000, (ref-fp64): 7283191157045600716183116389023744.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9048737Z E0605 10:21:04.904000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 367246206806026227203736201261481984.00000, (ref-fp64): 367246206806026227203736201261481984.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9053454Z E0605 10:21:04.904000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 116508796414857193940130948202889216.00000, (ref-fp64): 116508796414857193940130948202889216.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9057703Z E0605 10:21:04.905000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 57850197204253592612365040646881280.00000, (ref-fp64): 57850197204253592612365040646881280.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9062411Z E0605 10:21:04.905000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 86817738356569021038067826286395392.00000, (ref-fp64): 86817738356569021038067826286395392.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9066235Z E0605 10:21:04.906000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 15494925045445041486066401520123904.00000, (ref-fp64): 15494925045445041486066401520123904.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9072346Z E0605 10:21:04.906000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 838779910811816962625017749365063680.00000, (ref-fp64): 838779910811816962625017749365063680.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9076335Z E0605 10:21:04.907000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 7444475783777533327165203377291264.00000, (ref-fp64): 7444475783777533327165203377291264.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9080468Z E0605 10:21:04.907000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1248812984036432274371272665886359552.00000, (ref-fp64): 1248812984036432274371272665886359552.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9084644Z E0605 10:21:04.908000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 4435114980132863667434728909176832.00000, (ref-fp64): 4435114980132863667434728909176832.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9088951Z E0605 10:21:04.908000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 559820228411211407220176513165950976.00000, (ref-fp64): 559820228411211407220176513165950976.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9093799Z E0605 10:21:04.908000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 64776954020411122734398565353783296.00000, (ref-fp64): 64776954020411122734398565353783296.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9097985Z E0605 10:21:04.909000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 59136872147467286533966466897674240.00000, (ref-fp64): 59136872147467286533966466897674240.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9102942Z E0605 10:21:04.909000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 102243143291033031231876293556436992.00000, (ref-fp64): 102243143291033031231876293556436992.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9106649Z E0605 10:21:04.910000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 12255576975083271680881142622322688.00000, (ref-fp64): 12255576975083271680881142622322688.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9110918Z E0605 10:21:04.910000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1009944797429725682882241505131495424.00000, (ref-fp64): 1009944797429725682882241505131495424.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9114815Z E0605 10:21:04.911000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 2524273996166759519009086008983552.00000, (ref-fp64): 2524273996166759519009086008983552.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9118924Z E0605 10:21:04.911000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 616339573727521221107667254527393792.00000, (ref-fp64): 616339573727521221107667254527393792.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9123057Z E0605 10:21:04.911000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1060974210359713881391120240345088.00000, (ref-fp64): 1060974210359713881391120240345088.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9127388Z E0605 10:21:04.912000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 912756423786836159448964267419631616.00000, (ref-fp64): 912756423786836159448964267419631616.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9132096Z E0605 10:21:04.912000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 117838259954770006028208083627409408.00000, (ref-fp64): 117838259954770006028208083627409408.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9136460Z E0605 10:21:04.913000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 34943638735794087169027907266805760.00000, (ref-fp64): 34943638735794087169027907266805760.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9141071Z E0605 10:21:04.913000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 86624693115682291215518221717209088.00000, (ref-fp64): 86624693115682291215518221717209088.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9144809Z E0605 10:21:04.914000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 2169297399545674799976085970747392.00000, (ref-fp64): 2169297399545674799976085970747392.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9149135Z E0605 10:21:04.914000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 633988630609249091901742153541353472.00000, (ref-fp64): 633988630609249091901742153541353472.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9152988Z E0605 10:21:04.914000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1268440301637692580670415183020032.00000, (ref-fp64): 1268440301637692580670415183020032.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9157222Z E0605 10:21:04.915000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1477821457928288239496773205732884480.00000, (ref-fp64): 1477821457928288239496773205732884480.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9161143Z E0605 10:21:04.915000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 615350534034544314219815768162304.00000, (ref-fp64): 615350534034544314219815768162304.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9165865Z E0605 10:21:04.916000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 771594982641163879415112776462368768.00000, (ref-fp64): 771594982641163879415112776462368768.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9170658Z E0605 10:21:04.916000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 119403275880472927262329124866228224.00000, (ref-fp64): 119403275880472927262329124866228224.00000 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9175032Z E0605 10:21:04.917000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 97562760818130022182605709457227776.00000, (ref-fp64): 97562760818130022182605709457227776.00000 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9179752Z E0605 10:21:04.917000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 174672859975132121374601362903400448.00000, (ref-fp64): 174672859975132121374601362903400448.00000 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9183558Z E0605 10:21:04.917000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 706461476065345900879475986399232.00000, (ref-fp64): 706461476065345900879475986399232.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9187858Z E0605 10:21:04.918000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 517532112204890828039480189055401984.00000, (ref-fp64): 517532112204890828039480189055401984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9191706Z E0605 10:21:04.918000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 222461292486983289233846661283840.00000, (ref-fp64): 222461292486983289233846661283840.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9196000Z E0605 10:21:04.919000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 426301677534322155307646132349304832.00000, (ref-fp64): 426301677534322155307646132349304832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9200394Z E0605 10:21:04.919000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 68635978392252581495470985052160.00000, (ref-fp64): 68635978392252581495470985052160.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9204440Z E0605 10:21:04.919000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 265253250555013803016667409929994240.00000, (ref-fp64): 265253250555013803016667409929994240.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9209195Z E0605 10:21:04.920000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 65945373817317208445339207064354816.00000, (ref-fp64): 65945373817317208445339207064354816.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9213862Z E0605 10:21:04.920000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 13581638459375173440334339395026944.00000, (ref-fp64): 13581638459375173440334339395026944.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9219595Z E0605 10:21:04.921000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 27727670125527924333802392844763136.00000, (ref-fp64): 27727670125527924333802392844763136.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9224181Z E0605 10:21:04.921000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 8927210797347807355774630715457536.00000, (ref-fp64): 8927210797347807355774630715457536.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9228022Z E0605 10:21:04.922000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 68635978392252581495470985052160.00000, (ref-fp64): 68635978392252581495470985052160.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9231952Z E0605 10:21:04.922000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 58010647706461533743236822358556672.00000, (ref-fp64): 58010647706461533743236822358556672.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9235941Z E0605 10:21:04.923000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 304848586083316348671866748010496.00000, (ref-fp64): 304848586083316348671866748010496.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9240291Z E0605 10:21:04.923000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1206144284163441282965723595000512512.00000, (ref-fp64): 1206144284163441282965723595000512512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9244072Z E0605 10:21:04.923000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 95443683214690310004069202132992.00000, (ref-fp64): 95443683214690310004069202132992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9248675Z E0605 10:21:04.924000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 925052784207817554138016581844729856.00000, (ref-fp64): 925052784207817554138016581844729856.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9252669Z E0605 10:21:04.924000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 27109495725596663035477419360256.00000, (ref-fp64): 27109495725596663035477419360256.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9256535Z E0605 10:21:04.925000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 381220461837554643308257708200689664.00000, (ref-fp64): 381220461837554643308257708200689664.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9261163Z E0605 10:21:04.925000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 71976300051686215568156497046142976.00000, (ref-fp64): 71976300051686215568156497046142976.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9265918Z E0605 10:21:04.926000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 21591780189101318254947226091520000.00000, (ref-fp64): 21591780189101318254947226091520000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9270438Z E0605 10:21:04.926000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 46333750662755690005236733307781120.00000, (ref-fp64): 46333750662755690005236733307781120.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9273991Z E0605 10:21:04.927000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 2474599736247100274376704.00000, (ref-fp64): 2474599736247100274376704.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9278568Z E0605 10:21:04.927000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1225269526282335737781306320459137024.00000, (ref-fp64): 1225269526282335737781306320459137024.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9282175Z E0605 10:21:04.927000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 655413748481471274811392.00000, (ref-fp64): 655413748481471274811392.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9286967Z E0605 10:21:04.928000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 828317812917056754486388525470056448.00000, (ref-fp64): 828317812917056754486388525470056448.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9290660Z E0605 10:21:04.928000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 156864227812618264379392.00000, (ref-fp64): 156864227812618264379392.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9295064Z E0605 10:21:04.929000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 438037948125435551997376409722945536.00000, (ref-fp64): 438037948125435551997376409722945536.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9299691Z E0605 10:21:04.929000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 92040047188700659588450482210734080.00000, (ref-fp64): 92040047188700659588450482210734080.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9304276Z E0605 10:21:04.929000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 44361240799154524193240741413126144.00000, (ref-fp64): 44361240799154524193240741413126144.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9308841Z E0605 10:21:04.930000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 50579269376706930678194040377180160.00000, (ref-fp64): 50579269376706930678194040377180160.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9312479Z E0605 10:21:04.930000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 566198250110690116239360.00000, (ref-fp64): 566198250110690116239360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9317010Z E0605 10:21:04.931000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 995742051911006371484678381562232832.00000, (ref-fp64): 995742051911006371484678381562232832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9320646Z E0605 10:21:04.931000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 113604111401481771417600.00000, (ref-fp64): 113604111401481771417600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9325327Z E0605 10:21:04.932000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 733660191619024577931030104360615936.00000, (ref-fp64): 733660191619024577931030104360615936.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9329178Z E0605 10:21:04.932000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 46005128277734321029120.00000, (ref-fp64): 46005128277734321029120.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9333618Z E0605 10:21:04.932000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 704498631101713372003775477480161280.00000, (ref-fp64): 704498631101713372003775477480161280.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9337987Z E0605 10:21:04.933000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 54469216733055293823178867654787072.00000, (ref-fp64): 54469216733055293823178867654787072.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9342696Z E0605 10:21:04.933000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 27280233880705138902965888787415040.00000, (ref-fp64): 27280233880705138902965888787415040.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9347095Z E0605 10:21:04.934000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 55394899309495361132557864599027712.00000, (ref-fp64): 55394899309495361132557864599027712.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9350857Z E0605 10:21:04.934000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 187879122028730817445888.00000, (ref-fp64): 187879122028730817445888.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9355180Z E0605 10:21:04.935000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1381734679248587897684957777427955712.00000, (ref-fp64): 1381734679248587897684957777427955712.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9358877Z E0605 10:21:04.935000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 32180217928385935966208.00000, (ref-fp64): 32180217928385935966208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9363444Z E0605 10:21:04.935000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 963435701443446448904666328579702784.00000, (ref-fp64): 963435701443446448904666328579702784.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9367997Z E0605 10:21:04.936000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 5519711547670487105536.00000, (ref-fp64): 5519711547670487105536.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9372218Z E0605 10:21:04.936000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 431732016640994694131472074822123520.00000, (ref-fp64): 431732016640994694131472074822123520.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9376914Z E0605 10:21:04.937000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 88921601179249406953256010582589440.00000, (ref-fp64): 88921601179249406953256010582589440.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9381489Z E0605 10:21:04.937000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 35607071642181887121048545504788480.00000, (ref-fp64): 35607071642181887121048545504788480.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9386071Z E0605 10:21:04.938000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 44496614350694564403084532463435776.00000, (ref-fp64): 44496614350694564403084532463435776.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9389973Z E0605 10:21:04.938000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 18629489347757319651328.00000, (ref-fp64): 18629489347757319651328.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9394437Z E0605 10:21:04.939000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 958630534191807157784355126659514368.00000, (ref-fp64): 958630534191807157784355126659514368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9398394Z E0605 10:21:04.939000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 2770648093575851540480.00000, (ref-fp64): 2770648093575851540480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9402966Z E0605 10:21:04.939000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 706391102370083194830486013579624448.00000, (ref-fp64): 706391102370083194830486013579624448.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9407346Z E0605 10:21:04.940000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 625886425363475202048.00000, (ref-fp64): 625886425363475202048.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9411734Z E0605 10:21:04.940000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 681470407363910921569299468948340736.00000, (ref-fp64): 681470407363910921569299468948340736.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9416284Z E0605 10:21:04.941000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 53084364619663172562510951080787968.00000, (ref-fp64): 53084364619663172562510951080787968.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9420999Z E0605 10:21:04.941000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 22957133865035929587145674588160000.00000, (ref-fp64): 22957133865035929587145674588160000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9425533Z E0605 10:21:04.942000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 42873287532922945299439469304217600.00000, (ref-fp64): 42873287532922945299439469304217600.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9429034Z E0605 10:21:04.942000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1501933607326012145664.00000, (ref-fp64): 1501933607326012145664.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9433579Z E0605 10:21:04.942000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 987923881466771323795161423703703552.00000, (ref-fp64): 987923881466771323795161423703703552.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9437242Z E0605 10:21:04.943000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 197702848831546392576.00000, (ref-fp64): 197702848831546392576.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9441658Z E0605 10:21:04.943000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 917105577750224001164701132598018048.00000, (ref-fp64): 917105577750224001164701132598018048.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9445584Z E0605 10:21:04.944000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 40661187270111830016.00000, (ref-fp64): 40661187270111830016.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9449998Z E0605 10:21:04.944000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 448581091257288249725702204546875392.00000, (ref-fp64): 448581091257288249725702204546875392.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9455167Z E0605 10:21:04.945000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 49292150193612471862157293163380736.00000, (ref-fp64): 49292150193612471862157293163380736.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9460027Z E0605 10:21:04.945000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 23190320842352358250931895601725440.00000, (ref-fp64): 23190320842352358250931895601725440.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9464999Z E0605 10:21:04.946000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 44902841395700913060755062051045376.00000, (ref-fp64): 44902841395700913060755062051045376.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9469096Z E0605 10:21:04.946000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 94941130967120707584.00000, (ref-fp64): 94941130967120707584.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9473430Z E0605 10:21:04.946000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 778520048942101053340179781921013760.00000, (ref-fp64): 778520048942101053340179781921013760.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9477573Z E0605 10:21:04.947000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 15065452938550108160.00000, (ref-fp64): 15065452938550108160.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9482176Z E0605 10:21:04.947000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 993034499905156422407234477549944832.00000, (ref-fp64): 993034499905156422407234477549944832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9486831Z E0605 10:21:04.948000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 2991227293390027264.00000, (ref-fp64): 2991227293390027264.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9491343Z E0605 10:21:04.948000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 446805174238396735816339853005553664.00000, (ref-fp64): 446805174238396735816339853005553664.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9495759Z E0605 10:21:04.949000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 35820204745206290519354857790373888.00000, (ref-fp64): 35820204745206290519354857790373888.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9500967Z E0605 10:21:04.949000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 26242222371413852582163063278403584.00000, (ref-fp64): 26242222371413852582163063278403584.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9505279Z E0605 10:21:04.950000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 56423536738571717905616793715605504.00000, (ref-fp64): 56423536738571717905616793715605504.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9509004Z E0605 10:21:04.950000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 9761206042804131840.00000, (ref-fp64): 9761206042804131840.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9513734Z E0605 10:21:04.950000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 975638141319456272406927194837745664.00000, (ref-fp64): 975638141319456272406927194837745664.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9517890Z E0605 10:21:04.951000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 2083944040776746496.00000, (ref-fp64): 2083944040776746496.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9522300Z E0605 10:21:04.951000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1133471086851436008296782110119690240.00000, (ref-fp64): 1133471086851436008296782110119690240.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9526792Z E0605 10:21:04.952000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 515984119528470400.00000, (ref-fp64): 515984119528470400.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9531110Z E0605 10:21:04.952000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 649208901108355536401551728274571264.00000, (ref-fp64): 649208901108355536401551728274571264.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9535705Z E0605 10:21:04.953000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 56106304710352628114224589683294208.00000, (ref-fp64): 56106304710352628114224589683294208.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9540594Z E0605 10:21:04.953000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 36452953829377307577620644266770432.00000, (ref-fp64): 36452953829377307577620644266770432.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9545188Z E0605 10:21:04.954000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 73019984797269225180789433685770240.00000, (ref-fp64): 73019984797269225180789433685770240.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9549024Z E0605 10:21:04.954000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1838398151119407360.00000, (ref-fp64): 1838398151119407360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9553652Z E0605 10:21:04.954000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1065492841328928476940558892370231296.00000, (ref-fp64): 1065492841328928476940558892370231296.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9557632Z E0605 10:21:04.955000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 414687036166881984.00000, (ref-fp64): 414687036166881984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9562328Z E0605 10:21:04.955000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1178019208601108660090575123499712512.00000, (ref-fp64): 1178019208601108660090575123499712512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9566784Z E0605 10:21:04.956000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 150427894780387776.00000, (ref-fp64): 150427894780387776.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9571384Z E0605 10:21:04.956000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 833177903733437451018481684487077888.00000, (ref-fp64): 833177903733437451018481684487077888.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9576030Z E0605 10:21:04.957000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 52973457740586163834247506993086464.00000, (ref-fp64): 52973457740586163834247506993086464.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9581184Z E0605 10:21:04.957000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 47729855535099244851147699864469504.00000, (ref-fp64): 47729855535099244851147699864469504.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9585789Z E0605 10:21:04.958000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 118776410185792402377985466633814016.00000, (ref-fp64): 118776410185792402377985466633814016.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9589795Z E0605 10:21:04.958000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 705413340936593920.00000, (ref-fp64): 705413340936593920.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9594470Z E0605 10:21:04.959000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1219757937954221800532823331957637120.00000, (ref-fp64): 1219757937954221800532823331957637120.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9598884Z E0605 10:21:04.959000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 193442416441849600.00000, (ref-fp64): 193442416441849600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9603506Z E0605 10:21:04.959000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 741241646549820685960659492824678400.00000, (ref-fp64): 741241646549820685960659492824678400.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9608372Z E0605 10:21:04.960000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 48638192422010224.00000, (ref-fp64): 48638192422010224.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9613104Z E0605 10:21:04.960000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 489413419485197134227090713861947392.00000, (ref-fp64): 489413419485197134227090713861947392.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9617807Z E0605 10:21:04.961000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 119600955304016717915570365909696512.00000, (ref-fp64): 119600955304016717915570365909696512.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9622884Z E0605 10:21:04.961000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 41853413276882978127558039299948544.00000, (ref-fp64): 41853413276882978127558039299948544.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9627816Z E0605 10:21:04.962000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 55013110200598805910774958786609152.00000, (ref-fp64): 55013110200598805910774958786609152.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9631912Z E0605 10:21:04.962000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 148828611324839104.00000, (ref-fp64): 148828611324839104.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9636794Z E0605 10:21:04.963000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1026797735469344278901538450768396288.00000, (ref-fp64): 1026797735469344278901538450768396288.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9640826Z E0605 10:21:04.963000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 40524644879747912.00000, (ref-fp64): 40524644879747912.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9645815Z E0605 10:21:04.964000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 832414098244617706200207345175232512.00000, (ref-fp64): 832414098244617706200207345175232512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9650151Z E0605 10:21:04.964000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 11771326786965600.00000, (ref-fp64): 11771326786965600.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9654926Z E0605 10:21:04.965000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 803004091245259627808881791861784576.00000, (ref-fp64): 803004091245259627808881791861784576.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9659808Z E0605 10:21:04.965000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 61094237208805930101772212559151104.00000, (ref-fp64): 61094237208805930101772212559151104.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9664853Z E0605 10:21:04.966000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 38514193798506989283731591256866816.00000, (ref-fp64): 38514193798506989283731591256866816.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9669772Z E0605 10:21:04.966000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 64380402552830431977234677320646656.00000, (ref-fp64): 64380402552830431977234677320646656.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9674108Z E0605 10:21:04.966000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 183351060463415975201111264985088.00000, (ref-fp64): 183351060463415975201111264985088.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9678440Z E0605 10:21:04.967000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1651568865615648215723131605757198336.00000, (ref-fp64): 1651568865615648215723131605757198336.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9682661Z E0605 10:21:04.967000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 31306028635742872545287657750528.00000, (ref-fp64): 31306028635742872545287657750528.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9687496Z E0605 10:21:04.968000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 845760673502278505770706194879479808.00000, (ref-fp64): 845760673502278505770706194879479808.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9691795Z E0605 10:21:04.968000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 7815653392119151290662271320064.00000, (ref-fp64): 7815653392119151290662271320064.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9696169Z E0605 10:21:04.969000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 503246291415637058989309492480442368.00000, (ref-fp64): 503246291415637058989309492480442368.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9700998Z E0605 10:21:04.969000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 66483284859311995930426408611872768.00000, (ref-fp64): 66483284859311995930426408611872768.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9705976Z E0605 10:21:04.970000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 30259410123201918114895148097732608.00000, (ref-fp64): 30259410123201918114895148097732608.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9710978Z E0605 10:21:04.970000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 36762207743473848285779241807642624.00000, (ref-fp64): 36762207743473848285779241807642624.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9714599Z E0605 10:21:04.971000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 38708759614219424.00000, (ref-fp64): 38708759614219424.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9719391Z E0605 10:21:04.971000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1119694928764029322890348130285387776.00000, (ref-fp64): 1119694928764029322890348130285387776.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9724331Z E0605 10:21:04.971000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 9173009672628144.00000, (ref-fp64): 9173009672628144.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9729117Z E0605 10:21:04.972000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1336142673509459089650841675083284480.00000, (ref-fp64): 1336142673509459089650841675083284480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9732785Z E0605 10:21:04.972000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 2663392659093653.00000, (ref-fp64): 2663392659093653.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9737696Z E0605 10:21:04.973000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 958517668785251522010669197352239104.00000, (ref-fp64): 958517668785251522010669197352239104.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9742732Z E0605 10:21:04.973000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 81967179975210611069453154694201344.00000, (ref-fp64): 81967179975210611069453154694201344.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9747880Z E0605 10:21:04.974000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 48169369542672395147801828261888000.00000, (ref-fp64): 48169369542672395147801828261888000.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9752359Z E0605 10:21:04.974000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 103426182415423135533848038601654272.00000, (ref-fp64): 103426182415423135533848038601654272.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9756875Z E0605 10:21:04.975000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 11617456245545566.00000, (ref-fp64): 11617456245545566.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9761346Z E0605 10:21:04.975000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1073004649451466146832899563612798976.00000, (ref-fp64): 1073004649451466146832899563612798976.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9765776Z E0605 10:21:04.976000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 2839080274027609.00000, (ref-fp64): 2839080274027609.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9770258Z E0605 10:21:04.976000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 913972944520887533208392648982790144.00000, (ref-fp64): 913972944520887533208392648982790144.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9774519Z E0605 10:21:04.976000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 537520954548085.50000, (ref-fp64): 537520954548085.43750 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9779126Z E0605 10:21:04.977000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 447719658138651838802425476068409344.00000, (ref-fp64): 447719658138651838802425476068409344.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9783916Z E0605 10:21:04.977000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 113459886133572772625076154791886848.00000, (ref-fp64): 113459886133572772625076154791886848.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9788734Z E0605 10:21:04.978000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 48152616493888015549148856242405376.00000, (ref-fp64): 48152616493888015549148856242405376.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9793445Z E0605 10:21:04.978000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 50917674200265781616995395356852224.00000, (ref-fp64): 50917674200265781616995395356852224.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9797561Z E0605 10:21:04.979000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1815744959776403.00000, (ref-fp64): 1815744959776403.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9802213Z E0605 10:21:04.979000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 989668690078257713330478616223416320.00000, (ref-fp64): 989668690078257713330478616223416320.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9806872Z E0605 10:21:04.980000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 379030787751104.62500, (ref-fp64): 379030787751104.62500 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9811603Z E0605 10:21:04.980000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 859697196247783572340902634664755200.00000, (ref-fp64): 859697196247783572340902634664755200.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9815695Z E0605 10:21:04.981000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 63159319918890.42188, (ref-fp64): 63159319918890.39844 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9820271Z E0605 10:21:04.981000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 478751029349101887304998743082795008.00000, (ref-fp64): 478751029349101887304998743082795008.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9825072Z E0605 10:21:04.982000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 50070228967469299466156437384724480.00000, (ref-fp64): 50070228967469299466156437384724480.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9830153Z E0605 10:21:04.982000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 28032550318696006563373730909650944.00000, (ref-fp64): 28032550318696006563373730909650944.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9834756Z E0605 10:21:04.982000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 58321365317722918692766690628861952.00000, (ref-fp64): 58321365317722918692766690628861952.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9838639Z E0605 10:21:04.983000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 193084189623248.81250, (ref-fp64): 193084189623248.84375 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9843328Z E0605 10:21:04.983000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1130484205536902784987633352196489216.00000, (ref-fp64): 1130484205536902784987633352196489216.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9847883Z E0605 10:21:04.984000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 30108996070249.98828, (ref-fp64): 30108996070249.96484 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9852509Z E0605 10:21:04.984000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 833553742644266646942947969092026368.00000, (ref-fp64): 833553742644266646942947969092026368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9856608Z E0605 10:21:04.985000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 5239620925253.30273, (ref-fp64): 5239620925253.27246 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9861183Z E0605 10:21:04.985000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 419079920674163822196439831411687424.00000, (ref-fp64): 419079920674163822196439831411687424.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9865893Z E0605 10:21:04.986000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 60565356753394403593745789840523264.00000, (ref-fp64): 60565356753394403593745789840523264.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9870993Z E0605 10:21:04.986000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 32351185623669125922310435171729408.00000, (ref-fp64): 32351185623669125922310435171729408.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9875653Z E0605 10:21:04.987000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 47785593357169434395319729538465792.00000, (ref-fp64): 47785593357169434395319729538465792.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9879631Z E0605 10:21:04.987000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 16108271953228.61914, (ref-fp64): 16108271953228.61523 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9884509Z E0605 10:21:04.987000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1225041984349406491276679114834051072.00000, (ref-fp64): 1225041984349406491276679114834051072.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9888795Z E0605 10:21:04.988000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 3685134774439.16260, (ref-fp64): 3685134774439.20312 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9893329Z E0605 10:21:04.988000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1336152584040860449353210238286692352.00000, (ref-fp64): 1336152584040860449353210238286692352.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9897563Z E0605 10:21:04.989000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 615967102562.07471, (ref-fp64): 615967102562.05493 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9902118Z E0605 10:21:04.989000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 528011995069101775591010145381908480.00000, (ref-fp64): 528011995069101775591010145381908480.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9906808Z E0605 10:21:04.990000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 66010773133770402510588614920896512.00000, (ref-fp64): 66010773133770402510588614920896512.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9911783Z E0605 10:21:04.990000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 55990095013871970559257082687651840.00000, (ref-fp64): 55990095013871970559257082687651840.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9916548Z E0605 10:21:04.991000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 67062119026629051854210849738588160.00000, (ref-fp64): 67062119026629051854210849738588160.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9920778Z E0605 10:21:04.991000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1546600514630.27710, (ref-fp64): 1546600514630.20630 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9925784Z E0605 10:21:04.992000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 857670844378582966106880244566720512.00000, (ref-fp64): 857670844378582966106880244566720512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9929993Z E0605 10:21:04.992000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 257562494159.65189, (ref-fp64): 257562494159.55350 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9935007Z E0605 10:21:04.993000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 781013029975301092580172099561193472.00000, (ref-fp64): 781013029975301092580172099561193472.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9939213Z E0605 10:21:04.993000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 22772086977.34357, (ref-fp64): 22772086977.34040 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9944269Z E0605 10:21:04.993000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 321320504747965539223573830370852864.00000, (ref-fp64): 321320504747965539223573830370852864.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9948652Z E0605 10:21:04.994000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 43778329288644633905902831209021440.00000, (ref-fp64): 43778329288644633905902831209021440.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9953573Z E0605 10:21:04.994000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 28305912775687788272366026087202816.00000, (ref-fp64): 28305912775687788272366026087202816.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9958180Z E0605 10:21:04.995000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 31744759224723600387601605190483968.00000, (ref-fp64): 31744759224723600387601605190483968.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9962512Z E0605 10:21:04.995000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 51766933444.81186, (ref-fp64): 51766933444.79973 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9967496Z E0605 10:21:04.996000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 848328993195001007899427780312432640.00000, (ref-fp64): 848328993195001007899427780312432640.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9971568Z E0605 10:21:04.996000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 12468398298.43982, (ref-fp64): 12468398298.42087 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9976281Z E0605 10:21:04.997000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1158786466148279983908003978391060480.00000, (ref-fp64): 1158786466148279983908003978391060480.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9980509Z E0605 10:21:04.997000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 2817519624.24381, (ref-fp64): 2817519624.23086 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9984985Z E0605 10:21:04.998000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 678329179216136984250506922520739840.00000, (ref-fp64): 678329179216136984250506922520739840.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9989755Z E0605 10:21:04.998000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 31704828677486227262080960778207232.00000, (ref-fp64): 31704828677486227262080960778207232.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:04.9994836Z E0605 10:21:04.999000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 44091961357486099240958937238863872.00000, (ref-fp64): 44091961357486099240958937238863872.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:04.9999508Z E0605 10:21:04.999000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 90841863715252409928527808953319424.00000, (ref-fp64): 90841863715252409928527808953319424.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0003371Z E0605 10:21:04.999000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 8342620961.15644, (ref-fp64): 8342620961.12690 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0008568Z E0605 10:21:05.000000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1299849711782005396486631293643128832.00000, (ref-fp64): 1299849711782005396486631293643128832.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0012654Z E0605 10:21:05.000000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1618477502.91231, (ref-fp64): 1618477502.86645 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0017362Z E0605 10:21:05.001000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1444055492447421527083936775864844288.00000, (ref-fp64): 1444055492447421527083936775864844288.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0021606Z E0605 10:21:05.001000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 321148771.58314, (ref-fp64): 321148771.57406 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0026279Z E0605 10:21:05.002000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 596348652855786978467835232888291328.00000, (ref-fp64): 596348652855786978467835232888291328.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0030976Z E0605 10:21:05.002000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 77768920856811338407873453163020288.00000, (ref-fp64): 77768920856811338407873453163020288.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0036004Z E0605 10:21:05.003000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 49809114685426053514897328969875456.00000, (ref-fp64): 49809114685426053514897328969875456.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0040556Z E0605 10:21:05.003000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 79602655833809340158669535943065600.00000, (ref-fp64): 79602655833809340158669535943065600.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0044603Z E0605 10:21:05.004000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 799766708.91441, (ref-fp64): 799766708.87456 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0049637Z E0605 10:21:05.004000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 846330024107358438922443937638514688.00000, (ref-fp64): 846330024107358438922443937638514688.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0053695Z E0605 10:21:05.004000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 169017831.98850, (ref-fp64): 169017831.91391 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0058293Z E0605 10:21:05.005000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1051464272845668043296454504280489984.00000, (ref-fp64): 1051464272845668043296454504280489984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0062386Z E0605 10:21:05.005000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 45840474.59155, (ref-fp64): 45840474.59730 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0067735Z E0605 10:21:05.006000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 807657559490865540995749532944826368.00000, (ref-fp64): 807657559490865540995749532944826368.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0073374Z E0605 10:21:05.006000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 49963477635658453624141620338229248.00000, (ref-fp64): 49963477635658453624141620338229248.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0078487Z E0605 10:21:05.007000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 32982588733458169944398342039535616.00000, (ref-fp64): 32982588733458169944398342039535616.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0083319Z E0605 10:21:05.007000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 82848660824886751673043069630939136.00000, (ref-fp64): 82848660824886751673043069630939136.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0087880Z E0605 10:21:05.008000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 136144015.28997, (ref-fp64): 136144015.31543 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0102069Z E0605 10:21:05.008000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 736489110936374467106475597107298304.00000, (ref-fp64): 736489110936374467106475597107298304.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0105389Z E0605 10:21:05.009000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 27246041.41205, (ref-fp64): 27246041.44411 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0108692Z E0605 10:21:05.009000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 956364289562406020425371703429824512.00000, (ref-fp64): 956364289562406020425371703429824512.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0111807Z E0605 10:21:05.010000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 7966555.32747, (ref-fp64): 7966555.30873 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0115066Z E0605 10:21:05.010000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 776698031728042896515405538066956288.00000, (ref-fp64): 776698031728042896515405538066956288.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0118454Z E0605 10:21:05.010000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 74750701426893030622038086727499776.00000, (ref-fp64): 74750701426893030622038086727499776.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0121769Z E0605 10:21:05.011000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 39901043135052072775601839799271424.00000, (ref-fp64): 39901043135052072775601839799271424.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0125400Z E0605 10:21:05.011000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 79413297369165436433121067608834048.00000, (ref-fp64): 79413297369165436433121067608834048.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0128962Z E0605 10:21:05.012000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 22436867422799894648932026810368.00000, (ref-fp64): 22436867422799894648932026810368.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0133815Z E0605 10:21:05.012000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 838716291624339008926183523029614592.00000, (ref-fp64): 838716291624339008926183523029614592.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0137536Z E0605 10:21:05.013000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 3839113020754793980609804894208.00000, (ref-fp64): 3839113020754793980609804894208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0142524Z E0605 10:21:05.013000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 908533536220489530360839022401552384.00000, (ref-fp64): 908533536220489530360839022401552384.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0146821Z E0605 10:21:05.014000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1106567688013340808645491818496.00000, (ref-fp64): 1106567688013340808645491818496.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0151397Z E0605 10:21:05.014000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 458354600860767809764525159194034176.00000, (ref-fp64): 458354600860767809764525159194034176.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0156066Z E0605 10:21:05.015000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 46444137670002163791608214726901760.00000, (ref-fp64): 46444137670002163791608214726901760.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0161146Z E0605 10:21:05.015000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 17434873709154346729943278171455488.00000, (ref-fp64): 17434873709154346729943278171455488.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0165925Z E0605 10:21:05.016000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 45102900421403704639333530091913216.00000, (ref-fp64): 45102900421403704639333530091913216.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0170097Z E0605 10:21:05.016000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 25129012.28816, (ref-fp64): 25129012.29298 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0174815Z E0605 10:21:05.017000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 767687114747617552143811300445650944.00000, (ref-fp64): 767687114747617552143811300445650944.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0178890Z E0605 10:21:05.017000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 4865912.24516, (ref-fp64): 4865912.28430 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0183609Z E0605 10:21:05.017000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 988955017571954056124965876651786240.00000, (ref-fp64): 988955017571954056124965876651786240.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0187340Z E0605 10:21:05.018000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1643572.95967, (ref-fp64): 1643572.98497 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0192297Z E0605 10:21:05.018000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 594388958609800834331725869782925312.00000, (ref-fp64): 594388958609800834331725869782925312.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0197130Z E0605 10:21:05.019000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 55853359888312404266250002478661632.00000, (ref-fp64): 55853359888312404266250002478661632.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0202164Z E0605 10:21:05.019000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 35874698391258456971956689011998720.00000, (ref-fp64): 35874698391258456971956689011998720.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0207078Z E0605 10:21:05.020000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 80167853586892354628767043733684224.00000, (ref-fp64): 80167853586892354628767043733684224.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0211330Z E0605 10:21:05.020000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 5116232.90236, (ref-fp64): 5116232.90171 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0215979Z E0605 10:21:05.021000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 941668792588830439856081023031836672.00000, (ref-fp64): 941668792588830439856081023031836672.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0219719Z E0605 10:21:05.021000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1288700.40326, (ref-fp64): 1288700.46811 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0224562Z E0605 10:21:05.021000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1462815185603927402255617514205609984.00000, (ref-fp64): 1462815185603927402255617514205609984.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0228845Z E0605 10:21:05.022000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 310389.44665, (ref-fp64): 310389.45307 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0233451Z E0605 10:21:05.022000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 504266866468943654178810971851587584.00000, (ref-fp64): 504266866468943654178810971851587584.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0238076Z E0605 10:21:05.023000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 73041522803288847139799671058202624.00000, (ref-fp64): 73041522803288847139799671058202624.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0243144Z E0605 10:21:05.023000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 56823428535204210502773957740986368.00000, (ref-fp64): 56823428535204210502773957740986368.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0248311Z E0605 10:21:05.024000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 87248080098702910782807707379302400.00000, (ref-fp64): 87248080098702910782807707379302400.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0252260Z E0605 10:21:05.024000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 839922.03541, (ref-fp64): 839922.15530 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0256889Z E0605 10:21:05.025000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 882730838902755365147519514690191360.00000, (ref-fp64): 882730838902755365147519514690191360.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0261041Z E0605 10:21:05.025000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 158234.90175, (ref-fp64): 158234.99012 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0265782Z E0605 10:21:05.026000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 997322109187168842764469713539956736.00000, (ref-fp64): 997322109187168842764469713539956736.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0269764Z E0605 10:21:05.026000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 39573.14378, (ref-fp64): 39573.19076 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0274527Z E0605 10:21:05.026000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 722353475696903902373346618525089792.00000, (ref-fp64): 722353475696903902373346618525089792.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0279272Z E0605 10:21:05.027000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 65562810630065473372905447744339968.00000, (ref-fp64): 65562810630065473372905447744339968.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0284313Z E0605 10:21:05.027000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 40277082025714552882885854876401664.00000, (ref-fp64): 40277082025714552882885854876401664.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0289381Z E0605 10:21:05.028000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 75887356551688638296742540156600320.00000, (ref-fp64): 75887356551688638296742540156600320.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0293664Z E0605 10:21:05.028000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 91822.51183, (ref-fp64): 91822.58111 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0298003Z E0605 10:21:05.029000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 856724012486189534415817997283229696.00000, (ref-fp64): 856724012486189534415817997283229696.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0302078Z E0605 10:21:05.029000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 21733.26521, (ref-fp64): 21733.18570 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0306615Z E0605 10:21:05.030000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1226969190860660457565116106989895680.00000, (ref-fp64): 1226969190860660457565116106989895680.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0310749Z E0605 10:21:05.030000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 7868.80521, (ref-fp64): 7868.80868 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0315193Z E0605 10:21:05.031000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 673108421164364312818352424688287744.00000, (ref-fp64): 673108421164364312818352424688287744.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0319846Z E0605 10:21:05.031000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 66276701737543694419967538148409344.00000, (ref-fp64): 66276701737543694419967538148409344.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0324921Z E0605 10:21:05.031000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 45816052036161843796534322551324672.00000, (ref-fp64): 45816052036161843796534322551324672.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0329746Z E0605 10:21:05.032000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 136588162200073310329925485830602752.00000, (ref-fp64): 136588162200073310329925485830602752.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0333714Z E0605 10:21:05.032000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 22753.87108, (ref-fp64): 22753.83728 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0338243Z E0605 10:21:05.033000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1114580152243760967146404367917121536.00000, (ref-fp64): 1114580152243760967146404367917121536.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0342139Z E0605 10:21:05.033000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 4728.17098, (ref-fp64): 4728.18227 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0347010Z E0605 10:21:05.034000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 932305177095301615640120245216083968.00000, (ref-fp64): 932305177095301615640120245216083968.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0351032Z E0605 10:21:05.034000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1410.11579, (ref-fp64): 1410.11045 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0355635Z E0605 10:21:05.035000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 771624476355700286625720379113472000.00000, (ref-fp64): 771624476355700286625720379113472000.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0360454Z E0605 10:21:05.035000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 92808425232999125587556707732553728.00000, (ref-fp64): 92808425232999125587556707732553728.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0365773Z E0605 10:21:05.036000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 43799283046217300766220975819718656.00000, (ref-fp64): 43799283046217300766220975819718656.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0370581Z E0605 10:21:05.036000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 77902852386325130646031461427183616.00000, (ref-fp64): 77902852386325130646031461427183616.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0374674Z E0605 10:21:05.037000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 3412.10337, (ref-fp64): 3412.19220 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0379402Z E0605 10:21:05.037000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 803964599329983653860700547312517120.00000, (ref-fp64): 803964599329983653860700547312517120.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0383571Z E0605 10:21:05.037000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 794.08568, (ref-fp64): 794.17191 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0388166Z E0605 10:21:05.038000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 994053231901587030091598787594158080.00000, (ref-fp64): 994053231901587030091598787594158080.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0392256Z E0605 10:21:05.038000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 227.31336, (ref-fp64): 227.32922 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0397105Z E0605 10:21:05.039000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 806785814455352253055265390198784000.00000, (ref-fp64): 806785814455352253055265390198784000.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0401752Z E0605 10:21:05.039000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 70837816917400526837529608198291456.00000, (ref-fp64): 70837816917400526837529608198291456.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0407318Z E0605 10:21:05.040000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 41755892251916083752096834936373248.00000, (ref-fp64): 41755892251916083752096834936373248.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0412032Z E0605 10:21:05.040000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 100271412272271545253483265166147584.00000, (ref-fp64): 100271412272271545253483265166147584.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0416539Z E0605 10:21:05.041000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 2697084545334326633824834813952.00000, (ref-fp64): 2697084545334326633824834813952.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0420806Z E0605 10:21:05.041000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 806403064891016485067705442918989824.00000, (ref-fp64): 806403064891016485067705442918989824.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0425072Z E0605 10:21:05.042000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 402550930254345663095236460544.00000, (ref-fp64): 402550930254345663095236460544.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0429508Z E0605 10:21:05.042000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 770381274110288275223215600725655552.00000, (ref-fp64): 770381274110288275223215600725655552.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0433857Z E0605 10:21:05.042000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 65214877289651643402663493632.00000, (ref-fp64): 65214877289651643402663493632.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0438206Z E0605 10:21:05.043000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 403462996849529762860148872624209920.00000, (ref-fp64): 403462996849529762860148872624209920.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0442855Z E0605 10:21:05.043000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 39494949436539066375952744048492544.00000, (ref-fp64): 39494949436539066375952744048492544.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0448063Z E0605 10:21:05.044000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 19939703285302644774809206240837632.00000, (ref-fp64): 19939703285302644774809206240837632.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0452893Z E0605 10:21:05.044000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 40716726779866037392613748852654080.00000, (ref-fp64): 40716726779866037392613748852654080.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0456804Z E0605 10:21:05.045000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 208063969002249827363734945792.00000, (ref-fp64): 208063969002249827363734945792.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0461595Z E0605 10:21:05.045000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 871655981521272174727117236758118400.00000, (ref-fp64): 871655981521272174727117236758118400.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0465384Z E0605 10:21:05.046000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 42682298671877203270679134208.00000, (ref-fp64): 42682298671877203270679134208.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0470016Z E0605 10:21:05.046000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1045084248796786881273478946129707008.00000, (ref-fp64): 1045084248796786881273478946129707008.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0474318Z E0605 10:21:05.046000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 10225032162806111728822124544.00000, (ref-fp64): 10225032162806111728822124544.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0478733Z E0605 10:21:05.047000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 473848005397685379598959445767356416.00000, (ref-fp64): 473848005397685379598959445767356416.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0483853Z E0605 10:21:05.047000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 44587153138679443128252480832405504.00000, (ref-fp64): 44587153138679443128252480832405504.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0489343Z E0605 10:21:05.048000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 36944192896597056935129887964921856.00000, (ref-fp64): 36944192896597056935129887964921856.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0494275Z E0605 10:21:05.048000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 55146777983958046126810184789524480.00000, (ref-fp64): 55146777983958046126810184789524480.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0497987Z E0605 10:21:05.049000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 27443960825029650962309971968.00000, (ref-fp64): 27443960825029650962309971968.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0502783Z E0605 10:21:05.049000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1057206361871764728073585280201785344.00000, (ref-fp64): 1057206361871764728073585280201785344.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0507020Z E0605 10:21:05.050000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 4389239936061012311971725312.00000, (ref-fp64): 4389239936061012311971725312.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0511506Z E0605 10:21:05.050000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1038490380690512318490972941556121600.00000, (ref-fp64): 1038490380690512318490972941556121600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0515757Z E0605 10:21:05.051000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 628930793937539394934145024.00000, (ref-fp64): 628930793937539394934145024.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0520185Z E0605 10:21:05.051000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 369049690499509918795301773056671744.00000, (ref-fp64): 369049690499509918795301773056671744.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0525037Z E0605 10:21:05.052000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 42831236547709945689045120401276928.00000, (ref-fp64): 42831236547709945689045120401276928.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0530225Z E0605 10:21:05.052000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 23416171278680979267326554325123072.00000, (ref-fp64): 23416171278680979267326554325123072.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0535399Z E0605 10:21:05.053000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 43353681825085437334258265429639168.00000, (ref-fp64): 43353681825085437334258265429639168.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0539374Z E0605 10:21:05.053000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 2204634831438182270938120192.00000, (ref-fp64): 2204634831438182270938120192.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0544008Z E0605 10:21:05.053000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1097525900572439942348739020792528896.00000, (ref-fp64): 1097525900572439942348739020792528896.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0548073Z E0605 10:21:05.054000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 262889382673569719548641280.00000, (ref-fp64): 262889382673569719548641280.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0552561Z E0605 10:21:05.054000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 823076256447521401911075536585621504.00000, (ref-fp64): 823076256447521401911075536585621504.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0556712Z E0605 10:21:05.055000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 38172867593527516188377088.00000, (ref-fp64): 38172867593527516188377088.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0561263Z E0605 10:21:05.055000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 470387013579192995019915796693385216.00000, (ref-fp64): 470387013579192995019915796693385216.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0566325Z E0605 10:21:05.056000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 39014997152690412512455895256399872.00000, (ref-fp64): 39014997152690412512455895256399872.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0571696Z E0605 10:21:05.056000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 20015656584716282028025466582990848.00000, (ref-fp64): 20015656584716282028025466582990848.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0576112Z E0605 10:21:05.057000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 39773379167336401590236369664344064.00000, (ref-fp64): 39773379167336401590236369664344064.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0580421Z E0605 10:21:05.057000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 104615717539773008291299328.00000, (ref-fp64): 104615717539773008291299328.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0584832Z E0605 10:21:05.058000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1206399304361224419820770247196016640.00000, (ref-fp64): 1206399304361224419820770247196016640.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0589216Z E0605 10:21:05.058000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 12050829915725213691019264.00000, (ref-fp64): 12050829915725213691019264.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0593665Z E0605 10:21:05.058000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1006439677330553174814140065615183872.00000, (ref-fp64): 1006439677330553174814140065615183872.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0598117Z E0605 10:21:05.059000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 2247426468373113914523648.00000, (ref-fp64): 2247426468373113914523648.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0602543Z E0605 10:21:05.059000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 521985751320857161356998494943969280.00000, (ref-fp64): 521985751320857161356998494943969280.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0607558Z E0605 10:21:05.060000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 42322033481779692550167740641443840.00000, (ref-fp64): 42322033481779692550167740641443840.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0612513Z E0605 10:21:05.060000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 19362725474465412644163179046240256.00000, (ref-fp64): 19362725474465412644163179046240256.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0617409Z E0605 10:21:05.061000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 47645905660875351520089208111759360.00000, (ref-fp64): 47645905660875351520089208111759360.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0621700Z E0605 10:21:05.061000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 8290071516999408722903040.00000, (ref-fp64): 8290071516999408722903040.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0626318Z E0605 10:21:05.062000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1080726716816063182573558357915860992.00000, (ref-fp64): 1080726716816063182573558357915860992.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0630731Z E0605 10:21:05.062000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1720549850897383361085440.00000, (ref-fp64): 1720549850897383361085440.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0635245Z E0605 10:21:05.063000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 839552269389968249049173574320914432.00000, (ref-fp64): 839552269389968249049173574320914432.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0639486Z E0605 10:21:05.063000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 430574336249628205252608.00000, (ref-fp64): 430574336249628205252608.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0644244Z E0605 10:21:05.063000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 439625984212376172365757745155014656.00000, (ref-fp64): 439625984212376172365757745155014656.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0649140Z E0605 10:21:05.064000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 53772863595263445289631514840006656.00000, (ref-fp64): 53772863595263445289631514840006656.00000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0656132Z E0605 10:21:05.065000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 36270639141269013130725976711364608.00000, (ref-fp64): 36270639141269013130725976711364608.00000 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0660518Z E0605 10:21:05.065000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 62433524227627065809618512392486912.00000, (ref-fp64): 62433524227627065809618512392486912.00000 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0664545Z E0605 10:21:05.066000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 257.34189, (ref-fp64): 257.41995 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0669134Z E0605 10:21:05.066000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 345925374086881590424569056929513472.00000, (ref-fp64): 345925374086881590424569056929513472.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0673129Z E0605 10:21:05.066000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 38.78017, (ref-fp64): 38.76773 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0677781Z E0605 10:21:05.067000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 286027913821064859100809308487548928.00000, (ref-fp64): 286027913821064859100809308487548928.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0681756Z E0605 10:21:05.067000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1.59200, (ref-fp64): 1.59401 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0686906Z E0605 10:21:05.068000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 41074089939544335474297791919947776.00000, (ref-fp64): 41074089939544335474297791919947776.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0691931Z E0605 10:21:05.068000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 46406745618435589367546100972519424.00000, (ref-fp64): 46406745618435589367546100972519424.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0705506Z E0605 10:21:05.070000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 9133974275442376847740455400505344.00000, (ref-fp64): 9133974275442376847740455400505344.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0712125Z E0605 10:21:05.070000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 11647972941413937685548835998269440.00000, (ref-fp64): 11647972941413937685548835998269440.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0724130Z E0605 10:21:05.071000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 1342491099179038705368907552653312.00000, (ref-fp64): 1342491099179038705368907552653312.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0727773Z E0605 10:21:05.072000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 1.59200, (ref-fp64): 1.59401 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0732886Z E0605 10:21:05.072000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 4796739043464957014143572021084160.00000, (ref-fp64): 4796739043464957014143572021084160.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0736874Z E0605 10:21:05.073000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 8.08799, (ref-fp64): 8.13382 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0741558Z E0605 10:21:05.073000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 419717240651638446112849528612716544.00000, (ref-fp64): 419717240651638446112849528612716544.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0745864Z E0605 10:21:05.074000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1.89207, (ref-fp64): 1.93525 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0750578Z E0605 10:21:05.074000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 450090267235300963364664821780840448.00000, (ref-fp64): 450090267235300963364664821780840448.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0755154Z E0605 10:21:05.075000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.11370, (ref-fp64): 0.12348 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0759777Z E0605 10:21:05.075000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 138582444865557941846825831652917248.00000, (ref-fp64): 138582444865557941846825831652917248.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0766682Z E0605 10:21:05.076000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 13124673686888989399768932026417152.00000, (ref-fp64): 13124673686888989399768932026417152.00000 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0779038Z E0605 10:21:05.077000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 10633438256433389231171095278649344.00000, (ref-fp64): 10633438256433389231171095278649344.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0785620Z E0605 10:21:05.078000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 31189475496686633046139883952799744.00000, (ref-fp64): 31189475496686633046139883952799744.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0789460Z E0605 10:21:05.078000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 0.87699, (ref-fp64): 0.90747 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0794193Z E0605 10:21:05.078000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 975794775745105765840914670969421824.00000, (ref-fp64): 975794775745105765840914670969421824.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0797991Z E0605 10:21:05.079000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 0.24526, (ref-fp64): 0.24859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0802631Z E0605 10:21:05.079000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 455035582889973519510484832848183296.00000, (ref-fp64): 455035582889973519510484832848183296.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0807357Z E0605 10:21:05.080000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00638, (ref-fp64): 0.00555 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0812362Z E0605 10:21:05.080000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 16766609365298803616376214849060864.00000, (ref-fp64): 16766609365298803616376214849060864.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0819229Z E0605 10:21:05.081000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 44339628982455887873624647011401728.00000, (ref-fp64): 44339628982455887873624647011401728.00000 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0831626Z E0605 10:21:05.082000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 22597136556404823367199042280882176.00000, (ref-fp64): 22597136556404823367199042280882176.00000 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0838117Z E0605 10:21:05.083000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 15073195581965055047219014256295936.00000, (ref-fp64): 15073195581965055047219014256295936.00000 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0843779Z E0605 10:21:05.083000 140047412257408 torch/_dynamo/utils.py:1482] key: bn1.bias, passes_test: True, RMSE (res-fp64): 0.01327, (ref-fp64): 0.01358 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0848590Z E0605 10:21:05.084000 140047412257408 torch/_dynamo/utils.py:1482] key: bn1.weight, passes_test: True, RMSE (res-fp64): 0.01435, (ref-fp64): 0.01277 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0853925Z E0605 10:21:05.084000 140047412257408 torch/_dynamo/utils.py:1482] key: conv1.weight, passes_test: True, RMSE (res-fp64): 0.01050, (ref-fp64): 0.01051 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0867304Z E0605 10:21:05.086000 140047412257408 torch/_dynamo/utils.py:1482] key: fc.weight, passes_test: True, RMSE (res-fp64): 0.00201, (ref-fp64): 0.00309 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0872036Z E0605 10:21:05.086000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01064, (ref-fp64): 0.01049 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0876704Z E0605 10:21:05.087000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01053, (ref-fp64): 0.01015 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0881336Z E0605 10:21:05.087000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01251, (ref-fp64): 0.01283 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0886215Z E0605 10:21:05.088000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01150, (ref-fp64): 0.01188 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0891126Z E0605 10:21:05.088000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01159, (ref-fp64): 0.01300 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0895941Z E0605 10:21:05.089000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01032, (ref-fp64): 0.01158 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0900811Z E0605 10:21:05.089000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00883, (ref-fp64): 0.00858 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0905697Z E0605 10:21:05.090000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00771, (ref-fp64): 0.00742 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0910437Z E0605 10:21:05.090000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00876, (ref-fp64): 0.00965 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0915287Z E0605 10:21:05.091000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00965, (ref-fp64): 0.01057 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0919897Z E0605 10:21:05.091000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01159, (ref-fp64): 0.01300 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0924726Z E0605 10:21:05.092000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01145, (ref-fp64): 0.01185 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0929523Z E0605 10:21:05.092000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01029, (ref-fp64): 0.01110 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0934253Z E0605 10:21:05.092000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01040, (ref-fp64): 0.01387 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0939033Z E0605 10:21:05.093000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01240, (ref-fp64): 0.01111 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0943698Z E0605 10:21:05.093000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01151, (ref-fp64): 0.01236 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0948231Z E0605 10:21:05.094000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01282, (ref-fp64): 0.01420 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0952808Z E0605 10:21:05.094000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01162, (ref-fp64): 0.01239 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0957691Z E0605 10:21:05.095000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00779, (ref-fp64): 0.00817 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0962659Z E0605 10:21:05.095000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00808, (ref-fp64): 0.00764 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.0967779Z E0605 10:21:05.096000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00944, (ref-fp64): 0.01016 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0972420Z E0605 10:21:05.096000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00948, (ref-fp64): 0.01585 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0977170Z E0605 10:21:05.097000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01338 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0981753Z E0605 10:21:05.097000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01109, (ref-fp64): 0.01483 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0986510Z E0605 10:21:05.098000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01225, (ref-fp64): 0.01457 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0991123Z E0605 10:21:05.098000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01407, (ref-fp64): 0.01499 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.0995934Z E0605 10:21:05.099000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01294, (ref-fp64): 0.01384 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1000642Z E0605 10:21:05.099000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00862, (ref-fp64): 0.01184 and shape=torch.Size([64, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1005755Z E0605 10:21:05.100000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00940, (ref-fp64): 0.01034 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1010637Z E0605 10:21:05.100000 140047412257408 torch/_dynamo/utils.py:1482] key: layer1.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01102, (ref-fp64): 0.01133 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1015318Z E0605 10:21:05.101000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01459, (ref-fp64): 0.01521 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1019832Z E0605 10:21:05.101000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01418, (ref-fp64): 0.01630 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1024526Z E0605 10:21:05.102000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01630, (ref-fp64): 0.01671 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1029133Z E0605 10:21:05.102000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01553, (ref-fp64): 0.01608 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1033876Z E0605 10:21:05.102000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01330, (ref-fp64): 0.01431 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1038704Z E0605 10:21:05.103000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01206, (ref-fp64): 0.01353 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1043788Z E0605 10:21:05.103000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01142, (ref-fp64): 0.01253 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1049033Z E0605 10:21:05.104000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01084, (ref-fp64): 0.01151 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1054467Z E0605 10:21:05.105000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01046, (ref-fp64): 0.01098 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1059619Z E0605 10:21:05.105000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.01049, (ref-fp64): 0.01104 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1064631Z E0605 10:21:05.106000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01330, (ref-fp64): 0.01431 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1069571Z E0605 10:21:05.106000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01293, (ref-fp64): 0.01331 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1074657Z E0605 10:21:05.107000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01304, (ref-fp64): 0.01425 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1079548Z E0605 10:21:05.107000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01194, (ref-fp64): 0.01376 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1084745Z E0605 10:21:05.108000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01339, (ref-fp64): 0.01436 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1089848Z E0605 10:21:05.108000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01167, (ref-fp64): 0.01283 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1095057Z E0605 10:21:05.109000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01340, (ref-fp64): 0.01464 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1100031Z E0605 10:21:05.109000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01193, (ref-fp64): 0.01345 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1105640Z E0605 10:21:05.110000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00901, (ref-fp64): 0.00966 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1111026Z E0605 10:21:05.110000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00903, (ref-fp64): 0.00969 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1116382Z E0605 10:21:05.111000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00912, (ref-fp64): 0.00974 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1121202Z E0605 10:21:05.111000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01146, (ref-fp64): 0.01346 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1126833Z E0605 10:21:05.112000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01158, (ref-fp64): 0.01288 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1131936Z E0605 10:21:05.112000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01280, (ref-fp64): 0.01523 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1136979Z E0605 10:21:05.113000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01253, (ref-fp64): 0.01496 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1141950Z E0605 10:21:05.113000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01390, (ref-fp64): 0.01430 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1147063Z E0605 10:21:05.114000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01252, (ref-fp64): 0.01389 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1152455Z E0605 10:21:05.114000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00853, (ref-fp64): 0.00973 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1157810Z E0605 10:21:05.115000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00829, (ref-fp64): 0.00947 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1163583Z E0605 10:21:05.115000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00984, (ref-fp64): 0.01052 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1168802Z E0605 10:21:05.116000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01264, (ref-fp64): 0.01581 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1173879Z E0605 10:21:05.116000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01334, (ref-fp64): 0.01580 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1178819Z E0605 10:21:05.117000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01366, (ref-fp64): 0.01574 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1183632Z E0605 10:21:05.117000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01441, (ref-fp64): 0.01630 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1188724Z E0605 10:21:05.118000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01386, (ref-fp64): 0.01450 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1193709Z E0605 10:21:05.118000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01244, (ref-fp64): 0.01341 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1199254Z E0605 10:21:05.119000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00978, (ref-fp64): 0.01155 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1204777Z E0605 10:21:05.120000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00897, (ref-fp64): 0.01017 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1210492Z E0605 10:21:05.120000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.3.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00969, (ref-fp64): 0.01031 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1215438Z E0605 10:21:05.121000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01441, (ref-fp64): 0.01403 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1220543Z E0605 10:21:05.121000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01344, (ref-fp64): 0.01298 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1225447Z E0605 10:21:05.122000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01320, (ref-fp64): 0.01414 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1230590Z E0605 10:21:05.122000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01251, (ref-fp64): 0.01444 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1235492Z E0605 10:21:05.123000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01467, (ref-fp64): 0.01491 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1240657Z E0605 10:21:05.123000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01315, (ref-fp64): 0.01405 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1246299Z E0605 10:21:05.124000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01030, (ref-fp64): 0.01085 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1252278Z E0605 10:21:05.124000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00914, (ref-fp64): 0.00969 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1257498Z E0605 10:21:05.125000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.4.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00993, (ref-fp64): 0.01046 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1262722Z E0605 10:21:05.125000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01553, (ref-fp64): 0.01460 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1267456Z E0605 10:21:05.126000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01534, (ref-fp64): 0.01351 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1272349Z E0605 10:21:05.126000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01759, (ref-fp64): 0.01564 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1277193Z E0605 10:21:05.127000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01683, (ref-fp64): 0.01511 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1282225Z E0605 10:21:05.127000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01405, (ref-fp64): 0.01443 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1287610Z E0605 10:21:05.128000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01277, (ref-fp64): 0.01308 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1293730Z E0605 10:21:05.128000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01144, (ref-fp64): 0.01086 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1298622Z E0605 10:21:05.129000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01012, (ref-fp64): 0.00937 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1303984Z E0605 10:21:05.129000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.5.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00932, (ref-fp64): 0.00967 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1308940Z E0605 10:21:05.130000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01264, (ref-fp64): 0.01562 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1314139Z E0605 10:21:05.130000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01117, (ref-fp64): 0.01656 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1319032Z E0605 10:21:05.131000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01421, (ref-fp64): 0.01561 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1324029Z E0605 10:21:05.131000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01464, (ref-fp64): 0.01587 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1329208Z E0605 10:21:05.132000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01450, (ref-fp64): 0.01466 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1334301Z E0605 10:21:05.132000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01204, (ref-fp64): 0.01295 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1339521Z E0605 10:21:05.133000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00922, (ref-fp64): 0.01150 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1344834Z E0605 10:21:05.134000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00906, (ref-fp64): 0.01055 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1350096Z E0605 10:21:05.134000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.6.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00999, (ref-fp64): 0.01034 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1355027Z E0605 10:21:05.135000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01426, (ref-fp64): 0.01413 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1360505Z E0605 10:21:05.135000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01438, (ref-fp64): 0.01424 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1365061Z E0605 10:21:05.136000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01366, (ref-fp64): 0.01399 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1370044Z E0605 10:21:05.136000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01421, (ref-fp64): 0.01403 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1375172Z E0605 10:21:05.137000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01488, (ref-fp64): 0.01550 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1380003Z E0605 10:21:05.137000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01168, (ref-fp64): 0.01303 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1385424Z E0605 10:21:05.138000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01081, (ref-fp64): 0.01061 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1390821Z E0605 10:21:05.138000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00921, (ref-fp64): 0.00913 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1396132Z E0605 10:21:05.139000 140047412257408 torch/_dynamo/utils.py:1482] key: layer2.7.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00944, (ref-fp64): 0.00982 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1400801Z E0605 10:21:05.139000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01384, (ref-fp64): 0.01439 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1406187Z E0605 10:21:05.140000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01358, (ref-fp64): 0.01420 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1411220Z E0605 10:21:05.140000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01517, (ref-fp64): 0.01583 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1416063Z E0605 10:21:05.141000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01325, (ref-fp64): 0.01406 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1420979Z E0605 10:21:05.141000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01246, (ref-fp64): 0.01348 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1426016Z E0605 10:21:05.142000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01075, (ref-fp64): 0.01200 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1431223Z E0605 10:21:05.142000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01037, (ref-fp64): 0.01086 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1436593Z E0605 10:21:05.143000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00821, (ref-fp64): 0.00846 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1442110Z E0605 10:21:05.143000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00940, (ref-fp64): 0.01014 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1447920Z E0605 10:21:05.144000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00964, (ref-fp64): 0.01039 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1452702Z E0605 10:21:05.144000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01246, (ref-fp64): 0.01348 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1457664Z E0605 10:21:05.145000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01191, (ref-fp64): 0.01260 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1462427Z E0605 10:21:05.145000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01106, (ref-fp64): 0.01249 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1467427Z E0605 10:21:05.146000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01156, (ref-fp64): 0.01234 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1472347Z E0605 10:21:05.146000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01287, (ref-fp64): 0.01452 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1477278Z E0605 10:21:05.147000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01255, (ref-fp64): 0.01362 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1482239Z E0605 10:21:05.147000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01303, (ref-fp64): 0.01414 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1487547Z E0605 10:21:05.148000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01220, (ref-fp64): 0.01340 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1492831Z E0605 10:21:05.148000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00753, (ref-fp64): 0.00773 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1498360Z E0605 10:21:05.149000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00763, (ref-fp64): 0.00822 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1503364Z E0605 10:21:05.149000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00867, (ref-fp64): 0.00914 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1508280Z E0605 10:21:05.150000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01300, (ref-fp64): 0.01308 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1513248Z E0605 10:21:05.150000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01250, (ref-fp64): 0.01308 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1518030Z E0605 10:21:05.151000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01458, (ref-fp64): 0.01603 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1523249Z E0605 10:21:05.151000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01467, (ref-fp64): 0.01563 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1528560Z E0605 10:21:05.152000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01409, (ref-fp64): 0.01566 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1533813Z E0605 10:21:05.152000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01260, (ref-fp64): 0.01359 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1539166Z E0605 10:21:05.153000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00889, (ref-fp64): 0.00885 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1544436Z E0605 10:21:05.153000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00818, (ref-fp64): 0.00884 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1549676Z E0605 10:21:05.154000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.10.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00921, (ref-fp64): 0.00978 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1554533Z E0605 10:21:05.155000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01358, (ref-fp64): 0.01527 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1559478Z E0605 10:21:05.155000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01311, (ref-fp64): 0.01513 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1564528Z E0605 10:21:05.155000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01519, (ref-fp64): 0.01666 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1569762Z E0605 10:21:05.156000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01511, (ref-fp64): 0.01671 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1574705Z E0605 10:21:05.157000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01416, (ref-fp64): 0.01603 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1579676Z E0605 10:21:05.157000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01227, (ref-fp64): 0.01353 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1584970Z E0605 10:21:05.158000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00926, (ref-fp64): 0.01039 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1590533Z E0605 10:21:05.158000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00927, (ref-fp64): 0.01026 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1595934Z E0605 10:21:05.159000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.11.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00934, (ref-fp64): 0.01010 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1600958Z E0605 10:21:05.159000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01347, (ref-fp64): 0.01512 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1606181Z E0605 10:21:05.160000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01357, (ref-fp64): 0.01469 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1611415Z E0605 10:21:05.160000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01521, (ref-fp64): 0.01636 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1616193Z E0605 10:21:05.161000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01505, (ref-fp64): 0.01608 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1621046Z E0605 10:21:05.161000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01441, (ref-fp64): 0.01630 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1626053Z E0605 10:21:05.162000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01256, (ref-fp64): 0.01354 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1631336Z E0605 10:21:05.162000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00929, (ref-fp64): 0.01015 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1636781Z E0605 10:21:05.163000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00880, (ref-fp64): 0.00930 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1642044Z E0605 10:21:05.163000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.12.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00960, (ref-fp64): 0.01037 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1648791Z E0605 10:21:05.164000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01433, (ref-fp64): 0.01673 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1653663Z E0605 10:21:05.164000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01366, (ref-fp64): 0.01655 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1658558Z E0605 10:21:05.165000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01531, (ref-fp64): 0.01751 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1663475Z E0605 10:21:05.165000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01562, (ref-fp64): 0.01763 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1668401Z E0605 10:21:05.166000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01423, (ref-fp64): 0.01627 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1673401Z E0605 10:21:05.166000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01247, (ref-fp64): 0.01368 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1678784Z E0605 10:21:05.167000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00962, (ref-fp64): 0.01098 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1684310Z E0605 10:21:05.167000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00911, (ref-fp64): 0.01035 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1690736Z E0605 10:21:05.168000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.13.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00980, (ref-fp64): 0.01075 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1694957Z E0605 10:21:05.169000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01350, (ref-fp64): 0.01533 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1699961Z E0605 10:21:05.169000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01336, (ref-fp64): 0.01523 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1704994Z E0605 10:21:05.170000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01489, (ref-fp64): 0.01546 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1710090Z E0605 10:21:05.170000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01476, (ref-fp64): 0.01540 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1714982Z E0605 10:21:05.171000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01546, (ref-fp64): 0.01638 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1720006Z E0605 10:21:05.171000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01347, (ref-fp64): 0.01426 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1725395Z E0605 10:21:05.172000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00935, (ref-fp64): 0.01051 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1731010Z E0605 10:21:05.172000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00847, (ref-fp64): 0.00874 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1736249Z E0605 10:21:05.173000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.14.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00932, (ref-fp64): 0.00980 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1741149Z E0605 10:21:05.173000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01535, (ref-fp64): 0.01610 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1746020Z E0605 10:21:05.174000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01523, (ref-fp64): 0.01597 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1750912Z E0605 10:21:05.174000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01524, (ref-fp64): 0.01656 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1755966Z E0605 10:21:05.175000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01508, (ref-fp64): 0.01622 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1760896Z E0605 10:21:05.175000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01530, (ref-fp64): 0.01595 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1766089Z E0605 10:21:05.176000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01287, (ref-fp64): 0.01358 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1771398Z E0605 10:21:05.176000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01064, (ref-fp64): 0.01122 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1776763Z E0605 10:21:05.177000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00892, (ref-fp64): 0.00950 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1782333Z E0605 10:21:05.177000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.15.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00982, (ref-fp64): 0.01026 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1787078Z E0605 10:21:05.178000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01548, (ref-fp64): 0.01593 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1792218Z E0605 10:21:05.178000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01545, (ref-fp64): 0.01604 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1797206Z E0605 10:21:05.179000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01547, (ref-fp64): 0.01642 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1802360Z E0605 10:21:05.179000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01571, (ref-fp64): 0.01655 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1807772Z E0605 10:21:05.180000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01524, (ref-fp64): 0.01598 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1812671Z E0605 10:21:05.180000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01369, (ref-fp64): 0.01421 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1817908Z E0605 10:21:05.181000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01058, (ref-fp64): 0.01077 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1823274Z E0605 10:21:05.181000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00900, (ref-fp64): 0.00945 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1828426Z E0605 10:21:05.182000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.16.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00956, (ref-fp64): 0.00994 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1833320Z E0605 10:21:05.182000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01567, (ref-fp64): 0.01608 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1838273Z E0605 10:21:05.183000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01518, (ref-fp64): 0.01567 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1843245Z E0605 10:21:05.183000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01734, (ref-fp64): 0.01746 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1848531Z E0605 10:21:05.184000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01730, (ref-fp64): 0.01762 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1853653Z E0605 10:21:05.184000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01501, (ref-fp64): 0.01574 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1858448Z E0605 10:21:05.185000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01282, (ref-fp64): 0.01349 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1863987Z E0605 10:21:05.185000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01078, (ref-fp64): 0.01104 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1869304Z E0605 10:21:05.186000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00979, (ref-fp64): 0.00993 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1874861Z E0605 10:21:05.187000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.17.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00949, (ref-fp64): 0.00994 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1879630Z E0605 10:21:05.187000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01537, (ref-fp64): 0.01462 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1884994Z E0605 10:21:05.188000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01506, (ref-fp64): 0.01452 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1890129Z E0605 10:21:05.188000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01587, (ref-fp64): 0.01666 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1895053Z E0605 10:21:05.189000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01595, (ref-fp64): 0.01656 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1899972Z E0605 10:21:05.189000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01531, (ref-fp64): 0.01625 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1904883Z E0605 10:21:05.190000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01287, (ref-fp64): 0.01364 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1910340Z E0605 10:21:05.190000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00995, (ref-fp64): 0.00986 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1915648Z E0605 10:21:05.191000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00923, (ref-fp64): 0.00956 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1920913Z E0605 10:21:05.191000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.18.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00986, (ref-fp64): 0.01034 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1925905Z E0605 10:21:05.192000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01549, (ref-fp64): 0.01583 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1930899Z E0605 10:21:05.192000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01527, (ref-fp64): 0.01563 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1935834Z E0605 10:21:05.193000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01676, (ref-fp64): 0.01741 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1940744Z E0605 10:21:05.193000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01681, (ref-fp64): 0.01755 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1945803Z E0605 10:21:05.194000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01520, (ref-fp64): 0.01622 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1950912Z E0605 10:21:05.194000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01275, (ref-fp64): 0.01388 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1955765Z E0605 10:21:05.195000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01050, (ref-fp64): 0.01070 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1961296Z E0605 10:21:05.195000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.01008 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1966798Z E0605 10:21:05.196000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.19.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00985, (ref-fp64): 0.01051 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1971770Z E0605 10:21:05.196000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01475 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1976633Z E0605 10:21:05.197000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01328, (ref-fp64): 0.01384 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1981694Z E0605 10:21:05.197000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01757, (ref-fp64): 0.01790 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1986593Z E0605 10:21:05.198000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01734, (ref-fp64): 0.01774 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.1991595Z E0605 10:21:05.198000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01389, (ref-fp64): 0.01497 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.1996400Z E0605 10:21:05.199000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01180, (ref-fp64): 0.01268 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2001691Z E0605 10:21:05.199000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00926, (ref-fp64): 0.00929 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2007960Z E0605 10:21:05.200000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00976, (ref-fp64): 0.00959 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2013032Z E0605 10:21:05.200000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00910, (ref-fp64): 0.00950 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2017811Z E0605 10:21:05.201000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01523, (ref-fp64): 0.01657 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2022658Z E0605 10:21:05.201000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01476, (ref-fp64): 0.01629 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2027639Z E0605 10:21:05.202000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01639, (ref-fp64): 0.01740 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2032480Z E0605 10:21:05.202000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01633, (ref-fp64): 0.01745 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2037505Z E0605 10:21:05.203000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01528, (ref-fp64): 0.01625 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2042487Z E0605 10:21:05.203000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01316, (ref-fp64): 0.01388 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2048159Z E0605 10:21:05.204000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01020, (ref-fp64): 0.01101 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2053508Z E0605 10:21:05.204000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00935, (ref-fp64): 0.00978 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2058860Z E0605 10:21:05.205000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.20.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00965, (ref-fp64): 0.01009 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2063541Z E0605 10:21:05.205000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01473, (ref-fp64): 0.01635 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2068501Z E0605 10:21:05.206000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01629 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2073271Z E0605 10:21:05.206000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01676, (ref-fp64): 0.01754 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2078378Z E0605 10:21:05.207000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01654, (ref-fp64): 0.01749 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2083243Z E0605 10:21:05.207000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01582, (ref-fp64): 0.01672 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2088572Z E0605 10:21:05.208000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01356, (ref-fp64): 0.01436 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2093752Z E0605 10:21:05.208000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00964, (ref-fp64): 0.01052 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2099189Z E0605 10:21:05.209000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00905, (ref-fp64): 0.00950 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2104331Z E0605 10:21:05.210000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.21.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01007, (ref-fp64): 0.01055 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2109295Z E0605 10:21:05.210000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01635, (ref-fp64): 0.01694 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2114319Z E0605 10:21:05.211000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01585, (ref-fp64): 0.01639 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2119345Z E0605 10:21:05.211000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01673, (ref-fp64): 0.01801 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2124504Z E0605 10:21:05.212000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01658, (ref-fp64): 0.01758 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2129954Z E0605 10:21:05.212000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01568, (ref-fp64): 0.01670 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2134998Z E0605 10:21:05.213000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01359, (ref-fp64): 0.01437 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2140368Z E0605 10:21:05.213000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01092, (ref-fp64): 0.01134 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2145711Z E0605 10:21:05.214000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00960, (ref-fp64): 0.01011 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2150959Z E0605 10:21:05.214000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.22.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01040, (ref-fp64): 0.01088 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2155811Z E0605 10:21:05.215000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01535, (ref-fp64): 0.01605 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2160665Z E0605 10:21:05.215000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01555, (ref-fp64): 0.01591 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2165857Z E0605 10:21:05.216000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01758, (ref-fp64): 0.01713 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2170942Z E0605 10:21:05.216000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01760, (ref-fp64): 0.01724 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2175823Z E0605 10:21:05.217000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01612, (ref-fp64): 0.01673 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2180763Z E0605 10:21:05.217000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01395, (ref-fp64): 0.01461 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2185990Z E0605 10:21:05.218000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01036, (ref-fp64): 0.01096 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2191427Z E0605 10:21:05.218000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01005, (ref-fp64): 0.01014 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2196636Z E0605 10:21:05.219000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.23.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01054, (ref-fp64): 0.01095 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2201512Z E0605 10:21:05.219000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01540, (ref-fp64): 0.01624 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2206947Z E0605 10:21:05.220000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01561, (ref-fp64): 0.01648 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2211928Z E0605 10:21:05.220000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01705, (ref-fp64): 0.01731 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2217053Z E0605 10:21:05.221000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01723, (ref-fp64): 0.01732 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2222085Z E0605 10:21:05.221000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01651, (ref-fp64): 0.01725 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2226954Z E0605 10:21:05.222000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01428, (ref-fp64): 0.01529 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2232258Z E0605 10:21:05.222000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01083, (ref-fp64): 0.01121 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2237482Z E0605 10:21:05.223000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00973, (ref-fp64): 0.01006 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2242696Z E0605 10:21:05.223000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.24.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01056, (ref-fp64): 0.01101 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2248035Z E0605 10:21:05.224000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01779, (ref-fp64): 0.01866 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2253047Z E0605 10:21:05.224000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01779, (ref-fp64): 0.01872 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2257979Z E0605 10:21:05.225000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01858, (ref-fp64): 0.01864 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2262994Z E0605 10:21:05.225000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01859, (ref-fp64): 0.01854 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2268118Z E0605 10:21:05.226000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01663, (ref-fp64): 0.01723 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2273128Z E0605 10:21:05.226000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01464, (ref-fp64): 0.01520 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2278562Z E0605 10:21:05.227000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01234, (ref-fp64): 0.01293 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2284012Z E0605 10:21:05.227000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01030, (ref-fp64): 0.01043 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2289716Z E0605 10:21:05.228000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.25.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01075, (ref-fp64): 0.01115 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2294606Z E0605 10:21:05.229000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01748, (ref-fp64): 0.01758 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2299711Z E0605 10:21:05.229000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01729, (ref-fp64): 0.01742 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2304605Z E0605 10:21:05.230000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01742, (ref-fp64): 0.01732 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2309585Z E0605 10:21:05.230000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01744, (ref-fp64): 0.01740 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2314632Z E0605 10:21:05.231000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01556, (ref-fp64): 0.01612 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2319713Z E0605 10:21:05.231000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01338, (ref-fp64): 0.01417 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2325287Z E0605 10:21:05.232000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01231, (ref-fp64): 0.01237 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2330939Z E0605 10:21:05.232000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01040, (ref-fp64): 0.01059 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2336315Z E0605 10:21:05.233000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.26.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00990, (ref-fp64): 0.01030 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2341169Z E0605 10:21:05.233000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01585, (ref-fp64): 0.01605 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2346232Z E0605 10:21:05.234000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01570, (ref-fp64): 0.01576 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2351164Z E0605 10:21:05.234000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01687, (ref-fp64): 0.01631 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2356283Z E0605 10:21:05.235000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01659, (ref-fp64): 0.01580 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2361104Z E0605 10:21:05.235000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01561, (ref-fp64): 0.01638 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2366667Z E0605 10:21:05.236000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01304, (ref-fp64): 0.01407 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2372299Z E0605 10:21:05.236000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01092, (ref-fp64): 0.01097 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2377841Z E0605 10:21:05.237000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00969, (ref-fp64): 0.00984 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2382988Z E0605 10:21:05.237000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.27.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00983, (ref-fp64): 0.01034 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2387942Z E0605 10:21:05.238000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01705, (ref-fp64): 0.01792 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2392871Z E0605 10:21:05.238000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01665, (ref-fp64): 0.01725 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2397791Z E0605 10:21:05.239000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01808, (ref-fp64): 0.01813 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2402815Z E0605 10:21:05.239000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01792, (ref-fp64): 0.01795 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2408037Z E0605 10:21:05.240000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01484, (ref-fp64): 0.01583 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2413136Z E0605 10:21:05.240000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01304, (ref-fp64): 0.01378 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2418404Z E0605 10:21:05.241000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01161, (ref-fp64): 0.01203 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2423907Z E0605 10:21:05.241000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01064, (ref-fp64): 0.01081 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2429308Z E0605 10:21:05.242000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.28.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00974, (ref-fp64): 0.01029 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2434240Z E0605 10:21:05.242000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01404, (ref-fp64): 0.01545 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2439153Z E0605 10:21:05.243000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01407, (ref-fp64): 0.01529 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2444340Z E0605 10:21:05.243000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01493, (ref-fp64): 0.01563 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2449836Z E0605 10:21:05.244000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01495, (ref-fp64): 0.01568 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2454707Z E0605 10:21:05.245000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01524, (ref-fp64): 0.01594 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2459663Z E0605 10:21:05.245000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01309, (ref-fp64): 0.01372 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2465230Z E0605 10:21:05.246000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00959, (ref-fp64): 0.01043 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2470801Z E0605 10:21:05.246000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00867, (ref-fp64): 0.00922 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2476271Z E0605 10:21:05.247000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.29.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00949, (ref-fp64): 0.00987 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2481156Z E0605 10:21:05.247000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01506, (ref-fp64): 0.01601 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2486656Z E0605 10:21:05.248000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01461, (ref-fp64): 0.01561 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2491712Z E0605 10:21:05.248000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01521, (ref-fp64): 0.01561 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2496697Z E0605 10:21:05.249000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01486, (ref-fp64): 0.01520 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2501681Z E0605 10:21:05.249000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01373, (ref-fp64): 0.01527 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2506828Z E0605 10:21:05.250000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01210, (ref-fp64): 0.01291 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2512246Z E0605 10:21:05.250000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.01020 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2517574Z E0605 10:21:05.251000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00892, (ref-fp64): 0.00917 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2522877Z E0605 10:21:05.251000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.3.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00868, (ref-fp64): 0.00914 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2528209Z E0605 10:21:05.252000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01601 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2533087Z E0605 10:21:05.252000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01419, (ref-fp64): 0.01566 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2537930Z E0605 10:21:05.253000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01522, (ref-fp64): 0.01554 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2542732Z E0605 10:21:05.253000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01451, (ref-fp64): 0.01493 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2547678Z E0605 10:21:05.254000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01423, (ref-fp64): 0.01545 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2552634Z E0605 10:21:05.254000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01262, (ref-fp64): 0.01339 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2557977Z E0605 10:21:05.255000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01029, (ref-fp64): 0.01100 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2563385Z E0605 10:21:05.255000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00862, (ref-fp64): 0.00906 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2569057Z E0605 10:21:05.256000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.30.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00877, (ref-fp64): 0.00937 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2574305Z E0605 10:21:05.256000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01543 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2579156Z E0605 10:21:05.257000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01476, (ref-fp64): 0.01528 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2584029Z E0605 10:21:05.257000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01401, (ref-fp64): 0.01594 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2588853Z E0605 10:21:05.258000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01399, (ref-fp64): 0.01576 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2593830Z E0605 10:21:05.258000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01367, (ref-fp64): 0.01513 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2598770Z E0605 10:21:05.259000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01250, (ref-fp64): 0.01343 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2604188Z E0605 10:21:05.259000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00995, (ref-fp64): 0.01038 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2609887Z E0605 10:21:05.260000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00789, (ref-fp64): 0.00874 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2615439Z E0605 10:21:05.261000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.31.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00865, (ref-fp64): 0.00932 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2620223Z E0605 10:21:05.261000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01394, (ref-fp64): 0.01560 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2625118Z E0605 10:21:05.262000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01374, (ref-fp64): 0.01555 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2629915Z E0605 10:21:05.262000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01475, (ref-fp64): 0.01583 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2634902Z E0605 10:21:05.263000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01584 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2639724Z E0605 10:21:05.263000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.01527 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2645097Z E0605 10:21:05.264000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01245, (ref-fp64): 0.01343 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2650588Z E0605 10:21:05.264000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00953, (ref-fp64): 0.01054 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2656098Z E0605 10:21:05.265000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00807, (ref-fp64): 0.00889 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2661175Z E0605 10:21:05.265000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.32.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00906, (ref-fp64): 0.00970 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2666031Z E0605 10:21:05.266000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01445, (ref-fp64): 0.01473 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2670939Z E0605 10:21:05.266000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01436, (ref-fp64): 0.01431 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2675829Z E0605 10:21:05.267000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01432, (ref-fp64): 0.01481 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2680640Z E0605 10:21:05.267000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01421, (ref-fp64): 0.01481 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2685801Z E0605 10:21:05.268000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01419, (ref-fp64): 0.01507 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2690827Z E0605 10:21:05.268000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01282, (ref-fp64): 0.01370 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2696409Z E0605 10:21:05.269000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00942, (ref-fp64): 0.00984 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2701522Z E0605 10:21:05.269000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00734, (ref-fp64): 0.00796 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2706798Z E0605 10:21:05.270000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.33.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00815, (ref-fp64): 0.00874 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2711589Z E0605 10:21:05.270000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01362, (ref-fp64): 0.01389 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2716555Z E0605 10:21:05.271000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01383, (ref-fp64): 0.01393 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2721274Z E0605 10:21:05.271000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01502, (ref-fp64): 0.01621 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2726729Z E0605 10:21:05.272000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01601 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2731790Z E0605 10:21:05.272000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01420, (ref-fp64): 0.01546 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2736591Z E0605 10:21:05.273000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01252, (ref-fp64): 0.01374 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2741949Z E0605 10:21:05.273000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00896, (ref-fp64): 0.00939 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2747412Z E0605 10:21:05.274000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00734, (ref-fp64): 0.00821 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2752882Z E0605 10:21:05.274000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.34.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00881, (ref-fp64): 0.00953 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2757869Z E0605 10:21:05.275000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01594, (ref-fp64): 0.01747 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2762831Z E0605 10:21:05.275000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01583, (ref-fp64): 0.01708 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2768175Z E0605 10:21:05.276000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01533, (ref-fp64): 0.01575 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2773311Z E0605 10:21:05.276000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01509, (ref-fp64): 0.01572 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2778274Z E0605 10:21:05.277000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01414, (ref-fp64): 0.01517 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2783169Z E0605 10:21:05.277000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01227, (ref-fp64): 0.01365 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2788545Z E0605 10:21:05.278000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01024, (ref-fp64): 0.01121 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2793852Z E0605 10:21:05.278000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00827, (ref-fp64): 0.00923 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2799272Z E0605 10:21:05.279000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.35.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00847, (ref-fp64): 0.00927 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2804355Z E0605 10:21:05.279000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01522, (ref-fp64): 0.01815 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2809503Z E0605 10:21:05.280000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01495, (ref-fp64): 0.01773 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2814475Z E0605 10:21:05.281000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01546, (ref-fp64): 0.01661 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2819448Z E0605 10:21:05.281000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01536, (ref-fp64): 0.01652 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2824392Z E0605 10:21:05.281000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01412, (ref-fp64): 0.01559 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2829503Z E0605 10:21:05.282000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01211, (ref-fp64): 0.01318 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2834740Z E0605 10:21:05.283000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01173 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2840182Z E0605 10:21:05.283000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00954, (ref-fp64): 0.01005 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2845631Z E0605 10:21:05.284000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.4.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00951, (ref-fp64): 0.01020 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2850748Z E0605 10:21:05.284000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01221, (ref-fp64): 0.01540 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2855720Z E0605 10:21:05.285000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01202, (ref-fp64): 0.01527 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2860793Z E0605 10:21:05.285000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01429, (ref-fp64): 0.01628 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2865821Z E0605 10:21:05.286000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01413, (ref-fp64): 0.01599 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2870954Z E0605 10:21:05.286000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01405, (ref-fp64): 0.01563 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2876087Z E0605 10:21:05.287000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01157, (ref-fp64): 0.01285 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2881410Z E0605 10:21:05.287000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00885, (ref-fp64): 0.01071 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2887216Z E0605 10:21:05.288000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00787, (ref-fp64): 0.00878 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2892725Z E0605 10:21:05.288000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.5.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00892, (ref-fp64): 0.00958 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2897571Z E0605 10:21:05.289000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01327, (ref-fp64): 0.01461 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2902672Z E0605 10:21:05.289000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01296, (ref-fp64): 0.01455 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2907654Z E0605 10:21:05.290000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01519, (ref-fp64): 0.01607 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2912768Z E0605 10:21:05.290000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01531, (ref-fp64): 0.01599 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2917778Z E0605 10:21:05.291000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01486, (ref-fp64): 0.01555 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2922786Z E0605 10:21:05.291000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01296, (ref-fp64): 0.01336 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2928680Z E0605 10:21:05.292000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00895, (ref-fp64): 0.01006 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2934011Z E0605 10:21:05.292000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00861, (ref-fp64): 0.00885 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2939270Z E0605 10:21:05.293000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.6.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00948, (ref-fp64): 0.00982 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2944171Z E0605 10:21:05.293000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01392, (ref-fp64): 0.01436 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2949102Z E0605 10:21:05.294000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01379, (ref-fp64): 0.01387 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2954153Z E0605 10:21:05.294000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01696, (ref-fp64): 0.01681 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2959003Z E0605 10:21:05.295000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01688, (ref-fp64): 0.01639 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2964074Z E0605 10:21:05.295000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01499, (ref-fp64): 0.01565 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2969489Z E0605 10:21:05.296000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01279, (ref-fp64): 0.01373 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2975125Z E0605 10:21:05.297000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00941, (ref-fp64): 0.00953 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2980609Z E0605 10:21:05.297000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00959, (ref-fp64): 0.00929 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.2986027Z E0605 10:21:05.298000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.7.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00983, (ref-fp64): 0.01011 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2990851Z E0605 10:21:05.298000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01383, (ref-fp64): 0.01441 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.2996110Z E0605 10:21:05.299000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01374, (ref-fp64): 0.01413 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3000832Z E0605 10:21:05.299000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01345, (ref-fp64): 0.01394 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3006013Z E0605 10:21:05.300000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01322, (ref-fp64): 0.01359 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3011245Z E0605 10:21:05.300000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01421, (ref-fp64): 0.01530 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3016062Z E0605 10:21:05.301000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01233, (ref-fp64): 0.01328 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3021396Z E0605 10:21:05.301000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00954 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3026900Z E0605 10:21:05.302000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00759, (ref-fp64): 0.00771 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3032094Z E0605 10:21:05.302000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.8.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00866, (ref-fp64): 0.00906 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3036945Z E0605 10:21:05.303000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01410, (ref-fp64): 0.01520 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3042107Z E0605 10:21:05.303000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01385, (ref-fp64): 0.01490 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3047199Z E0605 10:21:05.304000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01515, (ref-fp64): 0.01727 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3052338Z E0605 10:21:05.304000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01493, (ref-fp64): 0.01698 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3057425Z E0605 10:21:05.305000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01385, (ref-fp64): 0.01494 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3062255Z E0605 10:21:05.305000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01205, (ref-fp64): 0.01311 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3067689Z E0605 10:21:05.306000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00936, (ref-fp64): 0.01000 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3073072Z E0605 10:21:05.306000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00834, (ref-fp64): 0.00900 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3078473Z E0605 10:21:05.307000 140047412257408 torch/_dynamo/utils.py:1482] key: layer3.9.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00908, (ref-fp64): 0.00948 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3083342Z E0605 10:21:05.307000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01309, (ref-fp64): 0.01405 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3088879Z E0605 10:21:05.308000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01331, (ref-fp64): 0.01428 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3093676Z E0605 10:21:05.308000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01303, (ref-fp64): 0.01336 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3098693Z E0605 10:21:05.309000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01287, (ref-fp64): 0.01324 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3103749Z E0605 10:21:05.309000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01178, (ref-fp64): 0.01107 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3108830Z E0605 10:21:05.310000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01071, (ref-fp64): 0.01016 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3114172Z E0605 10:21:05.310000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00854, (ref-fp64): 0.00926 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3127126Z E0605 10:21:05.312000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00660, (ref-fp64): 0.00687 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3133671Z E0605 10:21:05.312000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00784, (ref-fp64): 0.00821 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3145478Z E0605 10:21:05.314000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00802, (ref-fp64): 0.00864 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3150282Z E0605 10:21:05.314000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01178, (ref-fp64): 0.01107 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3155395Z E0605 10:21:05.315000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01076, (ref-fp64): 0.01042 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3160158Z E0605 10:21:05.315000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01074, (ref-fp64): 0.01222 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3165531Z E0605 10:21:05.316000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01064, (ref-fp64): 0.01209 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3170585Z E0605 10:21:05.316000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01151, (ref-fp64): 0.01261 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3175558Z E0605 10:21:05.317000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01119, (ref-fp64): 0.01224 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3180457Z E0605 10:21:05.317000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01173, (ref-fp64): 0.01094 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3185547Z E0605 10:21:05.318000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01068, (ref-fp64): 0.01002 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3192331Z E0605 10:21:05.318000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00587, (ref-fp64): 0.00739 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3205198Z E0605 10:21:05.320000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00446, (ref-fp64): 0.00597 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3211980Z E0605 10:21:05.320000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00680, (ref-fp64): 0.00804 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3216871Z E0605 10:21:05.321000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00906, (ref-fp64): 0.01037 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3221731Z E0605 10:21:05.321000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00902, (ref-fp64): 0.01032 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3226737Z E0605 10:21:05.322000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01304, (ref-fp64): 0.01424 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3231609Z E0605 10:21:05.322000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01194, (ref-fp64): 0.01329 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3236892Z E0605 10:21:05.323000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01163, (ref-fp64): 0.00921 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3241625Z E0605 10:21:05.323000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00982, (ref-fp64): 0.00914 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3248988Z E0605 10:21:05.324000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00502, (ref-fp64): 0.00629 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.3261355Z E0605 10:21:05.325000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00326, (ref-fp64): 0.00510 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:05.3267980Z E0605 10:21:05.326000 140047412257408 torch/_dynamo/utils.py:1482] key: layer4.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00683, (ref-fp64): 0.00786 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:21:05.4215875Z pass
2024-06-05T10:21:05.4588380Z TIMING: entire_frame_compile:155.36085 code_gen:56.70476 inductor_compile:102.79692 backend_compile:129.11051
2024-06-05T10:21:05.4589948Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:43385 | FakeTensorMode.__torch_dispatch__:242758 | ProxyTorchDispatchMode.__torch_dispatch__:53818
2024-06-05T10:21:05.4591169Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique)
2024-06-05T10:21:17.2002390Z 
2024-06-05T10:21:18.4386829Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:21:18.4387575Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:21:18.4388307Z cuda train resnet18                           
2024-06-05T10:21:34.3607384Z E0605 10:21:34.359000 139771622949504 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00166, (ref-fp64): 0.00165 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:21:34.4005549Z pass
2024-06-05T10:21:34.4076691Z TIMING: entire_frame_compile:7.59656 code_gen:5.57529 inductor_compile:8.37481 backend_compile:6.83519
2024-06-05T10:21:34.4078064Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:2590 | FakeTensorMode.__torch_dispatch__:15971 | ProxyTorchDispatchMode.__torch_dispatch__:4304
2024-06-05T10:21:34.4079271Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique)
2024-06-05T10:21:38.3908800Z 
2024-06-05T10:21:40.1675575Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:21:40.1676344Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:21:40.1677004Z cuda train resnet50                           
2024-06-05T10:22:11.2901368Z E0605 10:22:11.289000 140318894256768 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00205, (ref-fp64): 0.00208 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:22:11.3204214Z E0605 10:22:11.319000 140318894256768 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00000, (ref-fp64): 0.00007 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:22:11.3915544Z pass
2024-06-05T10:22:11.4076544Z TIMING: entire_frame_compile:14.44939 code_gen:9.55425 inductor_compile:15.90079 backend_compile:12.64674
2024-06-05T10:22:11.4078918Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:6704 | FakeTensorMode.__torch_dispatch__:40951 | ProxyTorchDispatchMode.__torch_dispatch__:11223
2024-06-05T10:22:11.4080143Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique)
2024-06-05T10:22:15.9569855Z 
2024-06-05T10:22:18.0871031Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:22:18.0871553Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:22:18.0872050Z cuda train resnet50_quantized_qat             
2024-06-05T10:22:18.0876590Z Traceback (most recent call last):
2024-06-05T10:22:18.0877705Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model
2024-06-05T10:22:18.0878631Z     self.model_iter_fn(model, example_inputs)
2024-06-05T10:22:18.0879747Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in forward_and_backward_pass
2024-06-05T10:22:18.0880995Z     pred = mod(*cloned_inputs)
2024-06-05T10:22:18.0882592Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 737, in call_wrapped
2024-06-05T10:22:18.0883908Z     return self._wrapped_call(self, *args, **kwargs)
2024-06-05T10:22:18.0885056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__
2024-06-05T10:22:18.0886223Z     raise e
2024-06-05T10:22:18.0887685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__
2024-06-05T10:22:18.0889044Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2024-06-05T10:22:18.0890496Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
2024-06-05T10:22:18.0891904Z     return self._call_impl(*args, **kwargs)
2024-06-05T10:22:18.0892938Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:22:18.0893850Z     return forward_call(*args, **kwargs)
2024-06-05T10:22:18.0894375Z   File "<eval_with_key>.3", line 167, in forward
2024-06-05T10:22:18.0895066Z     activation_post_process_73 = self.activation_post_process_73(fc);  fc = None
2024-06-05T10:22:18.0896303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
2024-06-05T10:22:18.0897265Z     return self._call_impl(*args, **kwargs)
2024-06-05T10:22:18.0898262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:22:18.0899271Z     return forward_call(*args, **kwargs)
2024-06-05T10:22:18.0900327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward
2024-06-05T10:22:18.0901319Z     return torch.fused_moving_avg_obs_fake_quant(
2024-06-05T10:22:18.0901930Z RuntimeError: expected scalar type Float but found Half
2024-06-05T10:22:18.0902335Z 
2024-06-05T10:22:18.0902655Z The above exception was the direct cause of the following exception:
2024-06-05T10:22:18.0903140Z 
2024-06-05T10:22:18.0903301Z Traceback (most recent call last):
2024-06-05T10:22:18.0904011Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T10:22:18.0904726Z     ) = runner.load_model(
2024-06-05T10:22:18.0905451Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model
2024-06-05T10:22:18.0906286Z     self.validate_model(model, example_inputs)
2024-06-05T10:22:18.0907119Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model
2024-06-05T10:22:18.0907959Z     raise RuntimeError("Eager run failed") from e
2024-06-05T10:22:18.0908471Z RuntimeError: Eager run failed
2024-06-05T10:22:18.0908749Z 
2024-06-05T10:22:18.0908877Z eager_fail_to_run
2024-06-05T10:22:21.3995591Z 
2024-06-05T10:22:23.2839828Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:22:23.2840351Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:22:23.2840841Z cuda train resnext50_32x4d                    
2024-06-05T10:22:55.0710919Z W0605 10:22:55.070000 140039933510272 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:23:38.6717180Z E0605 10:23:38.670000 140039933510272 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 37004721939964059648.00000, (ref-fp64): 37004721939964059648.00000 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6721200Z E0605 10:23:38.671000 140039933510272 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 36746346425896161280.00000, (ref-fp64): 36746346425896161280.00000 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6727194Z E0605 10:23:38.672000 140039933510272 torch/_dynamo/utils.py:1482] key: bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1080436394639282560.00000, (ref-fp64): 1080436394639282560.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6730913Z E0605 10:23:38.672000 140039933510272 torch/_dynamo/utils.py:1482] key: bn1.weight.grad, passes_test: True, RMSE (res-fp64): 2391393025224170496.00000, (ref-fp64): 2391393025224170496.00000 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6735769Z E0605 10:23:38.673000 140039933510272 torch/_dynamo/utils.py:1482] key: conv1.weight.grad, passes_test: True, RMSE (res-fp64): 24975794021394848.00000, (ref-fp64): 24975794021394848.00000 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6749166Z E0605 10:23:38.674000 140039933510272 torch/_dynamo/utils.py:1482] key: fc.weight.grad, passes_test: True, RMSE (res-fp64): 2071331881536352.50000, (ref-fp64): 2071331881536352.50000 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6753625Z E0605 10:23:38.674000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 794754405119088640.00000, (ref-fp64): 794754405119088640.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6758302Z E0605 10:23:38.675000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1693350396942251264.00000, (ref-fp64): 1693350396942251264.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6762617Z E0605 10:23:38.675000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 602331247833134464.00000, (ref-fp64): 602331247833134464.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6766998Z E0605 10:23:38.676000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1943391913961969920.00000, (ref-fp64): 1943391913961969920.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6771011Z E0605 10:23:38.676000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 307113351942385600.00000, (ref-fp64): 307113351942385600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6775151Z E0605 10:23:38.677000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 857761852659111808.00000, (ref-fp64): 857761852659111808.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6779519Z E0605 10:23:38.677000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 490743043255152448.00000, (ref-fp64): 490743043255152448.00000 and shape=torch.Size([128, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6783679Z E0605 10:23:38.677000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 655276351144342144.00000, (ref-fp64): 655276351144342144.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6788065Z E0605 10:23:38.678000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 199646472650708256.00000, (ref-fp64): 199646472650708256.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6792338Z E0605 10:23:38.678000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 297824915587809088.00000, (ref-fp64): 297824915587809088.00000 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6796420Z E0605 10:23:38.679000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 307113351942385600.00000, (ref-fp64): 307113351942385600.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6800646Z E0605 10:23:38.679000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 1181863427270611200.00000, (ref-fp64): 1181863427270611200.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6804866Z E0605 10:23:38.680000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 443324539170970304.00000, (ref-fp64): 443324539170970304.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6809513Z E0605 10:23:38.680000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 2081065677546305024.00000, (ref-fp64): 2081065677546305024.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6813662Z E0605 10:23:38.680000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 397735078670651584.00000, (ref-fp64): 397735078670651584.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6817770Z E0605 10:23:38.681000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1917868354062398720.00000, (ref-fp64): 1917868354062398720.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6821921Z E0605 10:23:38.681000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 144388017049485792.00000, (ref-fp64): 144388017049485792.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6826232Z E0605 10:23:38.682000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 1529807768992261120.00000, (ref-fp64): 1529807768992261120.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6830520Z E0605 10:23:38.682000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 322000321946047424.00000, (ref-fp64): 322000321946047424.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6834844Z E0605 10:23:38.683000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 822760044927529984.00000, (ref-fp64): 822760044927529984.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6839187Z E0605 10:23:38.683000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 221943892502712224.00000, (ref-fp64): 221943892502712224.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6843303Z E0605 10:23:38.683000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 229667840425162432.00000, (ref-fp64): 229667840425162432.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6847647Z E0605 10:23:38.684000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 3693263356514978816.00000, (ref-fp64): 3693263356514978816.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6851930Z E0605 10:23:38.684000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 198065659175187168.00000, (ref-fp64): 198065659175187168.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6855945Z E0605 10:23:38.685000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 2779137313339571200.00000, (ref-fp64): 2779137313339571200.00000 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6860019Z E0605 10:23:38.685000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 68545425084836776.00000, (ref-fp64): 68545425084836776.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6864156Z E0605 10:23:38.685000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 2308003723667874816.00000, (ref-fp64): 2308003723667874816.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6868637Z E0605 10:23:38.686000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 313442844223128448.00000, (ref-fp64): 313442844223128448.00000 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6872707Z E0605 10:23:38.686000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 1038780123696505600.00000, (ref-fp64): 1038780123696505600.00000 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6877079Z E0605 10:23:38.687000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 300870937660851264.00000, (ref-fp64): 300870937660851264.00000 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6880999Z E0605 10:23:38.687000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 58465902246591752.00000, (ref-fp64): 58465902246591760.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6885398Z E0605 10:23:38.688000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 2158304688806577152.00000, (ref-fp64): 2158304688806577152.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6889816Z E0605 10:23:38.688000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 18080039714265164.00000, (ref-fp64): 18080039714265164.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6893992Z E0605 10:23:38.688000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1919046935363044352.00000, (ref-fp64): 1919046935363044352.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6898102Z E0605 10:23:38.689000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 3978533496692100.50000, (ref-fp64): 3978533496692101.50000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6902298Z E0605 10:23:38.689000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 851251431773208704.00000, (ref-fp64): 851251431773208704.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6907087Z E0605 10:23:38.690000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 184407218608400960.00000, (ref-fp64): 184407218608400960.00000 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6911483Z E0605 10:23:38.690000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 243686482646839776.00000, (ref-fp64): 243686482646839776.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6916003Z E0605 10:23:38.691000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 89718028331538528.00000, (ref-fp64): 89718028331538528.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6920620Z E0605 10:23:38.691000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 11985594379755916.00000, (ref-fp64): 11985594379755916.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6924792Z E0605 10:23:38.692000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 3978533496692100.50000, (ref-fp64): 3978533496692101.50000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6929227Z E0605 10:23:38.692000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 144621713400583712.00000, (ref-fp64): 144621713400583712.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6933369Z E0605 10:23:38.692000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 6046748783186963.00000, (ref-fp64): 6046748783186964.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6937691Z E0605 10:23:38.693000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 3367463887796313088.00000, (ref-fp64): 3367463887796313088.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6941788Z E0605 10:23:38.693000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 4451585216033064.50000, (ref-fp64): 4451585216033065.50000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6946087Z E0605 10:23:38.694000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1491942169235417856.00000, (ref-fp64): 1491942169235417856.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6950199Z E0605 10:23:38.694000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1010410556368366.25000, (ref-fp64): 1010410556368366.87500 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6954533Z E0605 10:23:38.695000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 1539193095382635264.00000, (ref-fp64): 1539193095382635264.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6959070Z E0605 10:23:38.695000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 139664526067608800.00000, (ref-fp64): 139664526067608800.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6963504Z E0605 10:23:38.695000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 574000937150627392.00000, (ref-fp64): 574000937150627392.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.6968300Z E0605 10:23:38.696000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 151365062464393792.00000, (ref-fp64): 151365062464393792.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6972432Z E0605 10:23:38.696000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1472706805780054.00000, (ref-fp64): 1472706805780054.50000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6976662Z E0605 10:23:38.697000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 3329311946368118272.00000, (ref-fp64): 3329311946368118272.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6980790Z E0605 10:23:38.697000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 677497429327720.12500, (ref-fp64): 677497429327720.87500 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6984946Z E0605 10:23:38.698000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 2722597063803414528.00000, (ref-fp64): 2722597063803414528.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6989137Z E0605 10:23:38.698000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 183921100696933.28125, (ref-fp64): 183921100696933.81250 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6993280Z E0605 10:23:38.698000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 2198055149354711040.00000, (ref-fp64): 2198055149354711040.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.6997899Z E0605 10:23:38.699000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 166237407580071776.00000, (ref-fp64): 166237407580071776.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7002437Z E0605 10:23:38.699000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 522012777259312064.00000, (ref-fp64): 522012777259312064.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7007226Z E0605 10:23:38.700000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 200047070964461408.00000, (ref-fp64): 200047070964461408.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7011482Z E0605 10:23:38.700000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 304872881572424.56250, (ref-fp64): 304872881572425.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7015610Z E0605 10:23:38.701000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 2497688825515727872.00000, (ref-fp64): 2497688825515727872.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7019740Z E0605 10:23:38.701000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 213692290187449.21875, (ref-fp64): 213692290187450.31250 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7024044Z E0605 10:23:38.701000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 2102741266366877696.00000, (ref-fp64): 2102741266366877696.00000 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7028144Z E0605 10:23:38.702000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 43229586855358.65625, (ref-fp64): 43229586855359.50781 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7032567Z E0605 10:23:38.702000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 1557061734323560448.00000, (ref-fp64): 1557061734323560448.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7036974Z E0605 10:23:38.703000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 228817310693161344.00000, (ref-fp64): 228817310693161344.00000 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7041073Z E0605 10:23:38.703000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 632887190751524480.00000, (ref-fp64): 632887190751524480.00000 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7045953Z E0605 10:23:38.704000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 158302713278241216.00000, (ref-fp64): 158302713278241216.00000 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7049599Z E0605 10:23:38.704000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 22471290464040.28906, (ref-fp64): 22471290464041.24219 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7054047Z E0605 10:23:38.704000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1008565131207630208.00000, (ref-fp64): 1008565131207630208.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7058064Z E0605 10:23:38.705000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 6251494263396.18066, (ref-fp64): 6251494263397.01074 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7062223Z E0605 10:23:38.705000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1058424403032925824.00000, (ref-fp64): 1058424403032925824.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7066281Z E0605 10:23:38.706000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 968222367541.99707, (ref-fp64): 968222367542.38953 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7070339Z E0605 10:23:38.706000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 494376298606308096.00000, (ref-fp64): 494376298606308096.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7074906Z E0605 10:23:38.707000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 58192019844584008.00000, (ref-fp64): 58192019844584008.00000 and shape=torch.Size([512, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7079529Z E0605 10:23:38.707000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 92426079555688512.00000, (ref-fp64): 92426079555688512.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7084452Z E0605 10:23:38.707000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 33538693736376796.00000, (ref-fp64): 33538693736376796.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7089525Z E0605 10:23:38.708000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 2353690581278669.00000, (ref-fp64): 2353690581278669.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7093718Z E0605 10:23:38.708000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 968222367541.99707, (ref-fp64): 968222367542.38953 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7097729Z E0605 10:23:38.709000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 53089400135336416.00000, (ref-fp64): 53089400135336416.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7101892Z E0605 10:23:38.709000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 1491203056359.44604, (ref-fp64): 1491203056359.63037 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7106087Z E0605 10:23:38.710000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 2149823010016831744.00000, (ref-fp64): 2149823010016831744.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7110263Z E0605 10:23:38.710000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 665305150782.55579, (ref-fp64): 665305150783.04065 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7114503Z E0605 10:23:38.711000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1309722893007537152.00000, (ref-fp64): 1309722893007537152.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7118694Z E0605 10:23:38.711000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 104931357763.77969, (ref-fp64): 104931357764.16368 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7123033Z E0605 10:23:38.711000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 634470892746599808.00000, (ref-fp64): 634470892746599808.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7128165Z E0605 10:23:38.712000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 63738809478936456.00000, (ref-fp64): 63738809478936456.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7133004Z E0605 10:23:38.712000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 227711746360184288.00000, (ref-fp64): 227711746360184288.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7137552Z E0605 10:23:38.713000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 56884121822411568.00000, (ref-fp64): 56884121822411568.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7141635Z E0605 10:23:38.713000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 116756741058.48878, (ref-fp64): 116756741058.77849 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7146010Z E0605 10:23:38.714000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1682167624979401472.00000, (ref-fp64): 1682167624979401472.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7149896Z E0605 10:23:38.714000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 51065005951.00156, (ref-fp64): 51065005951.63967 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7154255Z E0605 10:23:38.714000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1273156281389955328.00000, (ref-fp64): 1273156281389955328.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7158416Z E0605 10:23:38.715000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 6727927213.71856, (ref-fp64): 6727927214.12361 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7162725Z E0605 10:23:38.715000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 756402576371474048.00000, (ref-fp64): 756402576371474048.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7167874Z E0605 10:23:38.716000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 58679691619189656.00000, (ref-fp64): 58679691619189656.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7172667Z E0605 10:23:38.716000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 236708042528294144.00000, (ref-fp64): 236708042528294144.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7177179Z E0605 10:23:38.717000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 54049270738605104.00000, (ref-fp64): 54049270738605104.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7181100Z E0605 10:23:38.717000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 5877575925.96700, (ref-fp64): 5877575926.27232 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7185330Z E0605 10:23:38.718000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1530254061752436480.00000, (ref-fp64): 1530254061752436480.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7189326Z E0605 10:23:38.718000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 2215092023.41601, (ref-fp64): 2215092024.02404 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7193332Z E0605 10:23:38.718000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1440625416364438784.00000, (ref-fp64): 1440625416364438784.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7197451Z E0605 10:23:38.719000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 313575431.88041, (ref-fp64): 313575432.32671 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7201645Z E0605 10:23:38.719000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 944048435654065664.00000, (ref-fp64): 944048435654065664.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7206879Z E0605 10:23:38.720000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 52994548550202128.00000, (ref-fp64): 52994548550202128.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7211551Z E0605 10:23:38.720000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 211774121547048704.00000, (ref-fp64): 211774121547048704.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7216010Z E0605 10:23:38.721000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 57676302636389480.00000, (ref-fp64): 57676302636389480.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7219673Z E0605 10:23:38.721000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 292582225.01367, (ref-fp64): 292582225.25993 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7224042Z E0605 10:23:38.721000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1582117620286287616.00000, (ref-fp64): 1582117620286287616.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7227878Z E0605 10:23:38.722000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 92041785.27299, (ref-fp64): 92041785.95236 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7232087Z E0605 10:23:38.722000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1621857740286198528.00000, (ref-fp64): 1621857740286198528.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7236129Z E0605 10:23:38.723000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 13134803.41969, (ref-fp64): 13134803.84750 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7240279Z E0605 10:23:38.723000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 963462929634031872.00000, (ref-fp64): 963462929634031872.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7245023Z E0605 10:23:38.724000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 58619612462221784.00000, (ref-fp64): 58619612462221784.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7250024Z E0605 10:23:38.724000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 196313275562248064.00000, (ref-fp64): 196313275562248064.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7254440Z E0605 10:23:38.724000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 67835089144001808.00000, (ref-fp64): 67835089144001808.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7258462Z E0605 10:23:38.725000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 11654732.86079, (ref-fp64): 11654733.01088 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7262492Z E0605 10:23:38.725000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 1428533944575174144.00000, (ref-fp64): 1428533944575174144.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7266534Z E0605 10:23:38.726000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 3791433.67124, (ref-fp64): 3791434.38002 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7270861Z E0605 10:23:38.726000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 1375826149955597056.00000, (ref-fp64): 1375826149955597056.00000 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7274947Z E0605 10:23:38.727000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 776309.33985, (ref-fp64): 776310.00533 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7279308Z E0605 10:23:38.727000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 1186998677293331456.00000, (ref-fp64): 1186998677293331456.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7284099Z E0605 10:23:38.727000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 57504690394041480.00000, (ref-fp64): 57504690394041480.00000 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7288949Z E0605 10:23:38.728000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 199674809046237664.00000, (ref-fp64): 199674809046237664.00000 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7293678Z E0605 10:23:38.728000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 77978445266419568.00000, (ref-fp64): 77978445266419568.00000 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7297613Z E0605 10:23:38.729000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 262256.69502, (ref-fp64): 262257.66713 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7302062Z E0605 10:23:38.729000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 603019909294110720.00000, (ref-fp64): 603019909294110720.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7306017Z E0605 10:23:38.730000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 29506.49415, (ref-fp64): 29506.86530 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7310429Z E0605 10:23:38.730000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 501014250606529600.00000, (ref-fp64): 501014250606529600.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7314639Z E0605 10:23:38.731000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 1848.00666, (ref-fp64): 1848.23230 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7319054Z E0605 10:23:38.731000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 154978656861878240.00000, (ref-fp64): 154978656861878240.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7327552Z E0605 10:23:38.732000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 24182921609577900.00000, (ref-fp64): 24182921609577900.00000 and shape=torch.Size([1024, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7332138Z E0605 10:23:38.732000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 35991615444811628.00000, (ref-fp64): 35991615444811628.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7343303Z E0605 10:23:38.733000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 9467841050185580.00000, (ref-fp64): 9467841050185580.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7354573Z E0605 10:23:38.735000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 320349294506987.25000, (ref-fp64): 320349294506987.25000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7358550Z E0605 10:23:38.735000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.bias.grad, passes_test: True, RMSE (res-fp64): 1848.00666, (ref-fp64): 1848.23230 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7362997Z E0605 10:23:38.735000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 7267959725199597.00000, (ref-fp64): 7267959725199597.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7367256Z E0605 10:23:38.736000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 791.26613, (ref-fp64): 791.76398 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7371831Z E0605 10:23:38.736000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 689592714139355776.00000, (ref-fp64): 689592714139355776.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7375811Z E0605 10:23:38.737000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 235.86755, (ref-fp64): 236.37316 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7380206Z E0605 10:23:38.737000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 658727893153799296.00000, (ref-fp64): 658727893153799296.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7384202Z E0605 10:23:38.738000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 13.20530, (ref-fp64): 13.45803 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7388802Z E0605 10:23:38.738000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 216863613111810688.00000, (ref-fp64): 216863613111810688.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7400183Z E0605 10:23:38.739000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 14120995187053598.00000, (ref-fp64): 14120995187053598.00000 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7404646Z E0605 10:23:38.740000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 63704817458537232.00000, (ref-fp64): 63704817458537232.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7416007Z E0605 10:23:38.741000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 14093571675526578.00000, (ref-fp64): 14093571675526578.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7419883Z E0605 10:23:38.741000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 13.67219, (ref-fp64): 14.23236 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7424006Z E0605 10:23:38.741000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 725694717256641920.00000, (ref-fp64): 725694717256641920.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7428040Z E0605 10:23:38.742000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.bias.grad, passes_test: True, RMSE (res-fp64): 1.21527, (ref-fp64): 1.33348 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7432163Z E0605 10:23:38.742000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.weight.grad, passes_test: True, RMSE (res-fp64): 562045303027117440.00000, (ref-fp64): 562045303027117440.00000 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7436163Z E0605 10:23:38.743000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00401, (ref-fp64): 0.00495 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7440475Z E0605 10:23:38.743000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 25408374642376072.00000, (ref-fp64): 25408374642376072.00000 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7451944Z E0605 10:23:38.744000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 15408686861042556.00000, (ref-fp64): 15408686861042556.00000 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7456087Z E0605 10:23:38.745000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 51885259565403408.00000, (ref-fp64): 51885259565403408.00000 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7467538Z E0605 10:23:38.746000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 6382744681132833.00000, (ref-fp64): 6382744681132833.00000 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7472223Z E0605 10:23:38.746000 140039933510272 torch/_dynamo/utils.py:1482] key: bn1.bias, passes_test: True, RMSE (res-fp64): 0.01006, (ref-fp64): 0.00822 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7476534Z E0605 10:23:38.747000 140039933510272 torch/_dynamo/utils.py:1482] key: bn1.weight, passes_test: True, RMSE (res-fp64): 0.01073, (ref-fp64): 0.00841 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7481184Z E0605 10:23:38.747000 140039933510272 torch/_dynamo/utils.py:1482] key: conv1.weight, passes_test: True, RMSE (res-fp64): 0.01076, (ref-fp64): 0.00786 and shape=torch.Size([64, 3, 7, 7]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7494803Z E0605 10:23:38.749000 140039933510272 torch/_dynamo/utils.py:1482] key: fc.weight, passes_test: True, RMSE (res-fp64): 0.00176, (ref-fp64): 0.00267 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7498997Z E0605 10:23:38.749000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00791, (ref-fp64): 0.00648 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7503362Z E0605 10:23:38.749000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00795, (ref-fp64): 0.00675 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7507611Z E0605 10:23:38.750000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01273, (ref-fp64): 0.00824 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7512086Z E0605 10:23:38.750000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01042, (ref-fp64): 0.00777 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7516262Z E0605 10:23:38.751000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01074, (ref-fp64): 0.00833 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7520581Z E0605 10:23:38.751000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01026, (ref-fp64): 0.00805 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7525397Z E0605 10:23:38.752000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00759, (ref-fp64): 0.00606 and shape=torch.Size([128, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7529997Z E0605 10:23:38.752000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00837, (ref-fp64): 0.00625 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7534683Z E0605 10:23:38.753000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00923, (ref-fp64): 0.00716 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7538925Z E0605 10:23:38.753000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00972, (ref-fp64): 0.00755 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7543275Z E0605 10:23:38.753000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01074, (ref-fp64): 0.00833 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7547908Z E0605 10:23:38.754000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01029, (ref-fp64): 0.00805 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7552234Z E0605 10:23:38.754000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01138, (ref-fp64): 0.00731 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7556667Z E0605 10:23:38.755000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01179, (ref-fp64): 0.00729 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7560980Z E0605 10:23:38.755000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01143, (ref-fp64): 0.00846 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7565751Z E0605 10:23:38.756000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01082, (ref-fp64): 0.00849 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7570270Z E0605 10:23:38.756000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01197, (ref-fp64): 0.00893 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7574672Z E0605 10:23:38.757000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01217, (ref-fp64): 0.00895 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7579539Z E0605 10:23:38.757000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00833, (ref-fp64): 0.00604 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7584349Z E0605 10:23:38.757000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00991, (ref-fp64): 0.00716 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7588820Z E0605 10:23:38.758000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00993, (ref-fp64): 0.00762 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7593280Z E0605 10:23:38.758000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01105, (ref-fp64): 0.00887 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7597447Z E0605 10:23:38.759000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01063, (ref-fp64): 0.00863 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7601902Z E0605 10:23:38.759000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01040, (ref-fp64): 0.00894 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7606529Z E0605 10:23:38.760000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01062, (ref-fp64): 0.00898 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7610893Z E0605 10:23:38.760000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01188, (ref-fp64): 0.00912 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7615191Z E0605 10:23:38.761000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01188, (ref-fp64): 0.00880 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7619834Z E0605 10:23:38.761000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00981, (ref-fp64): 0.00738 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7624175Z E0605 10:23:38.762000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01006, (ref-fp64): 0.00826 and shape=torch.Size([128, 4, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7628709Z E0605 10:23:38.762000 140039933510272 torch/_dynamo/utils.py:1482] key: layer1.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01055, (ref-fp64): 0.00790 and shape=torch.Size([256, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7632937Z E0605 10:23:38.762000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01316, (ref-fp64): 0.00979 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7637294Z E0605 10:23:38.763000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01283, (ref-fp64): 0.00944 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7641583Z E0605 10:23:38.763000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01242, (ref-fp64): 0.00924 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7646637Z E0605 10:23:38.764000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01164, (ref-fp64): 0.00887 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7651095Z E0605 10:23:38.764000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01242, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7655600Z E0605 10:23:38.765000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01218, (ref-fp64): 0.00867 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7660430Z E0605 10:23:38.765000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01151, (ref-fp64): 0.00824 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7665004Z E0605 10:23:38.766000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01093, (ref-fp64): 0.00763 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7669860Z E0605 10:23:38.766000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.00732 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7674625Z E0605 10:23:38.767000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00996, (ref-fp64): 0.00722 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7679197Z E0605 10:23:38.767000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01242, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7683859Z E0605 10:23:38.767000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01196, (ref-fp64): 0.00865 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7688593Z E0605 10:23:38.768000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01150, (ref-fp64): 0.00920 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7692993Z E0605 10:23:38.768000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01147, (ref-fp64): 0.00840 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7697481Z E0605 10:23:38.769000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01182, (ref-fp64): 0.00956 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7701849Z E0605 10:23:38.769000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01191, (ref-fp64): 0.00947 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7706195Z E0605 10:23:38.770000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01212, (ref-fp64): 0.00942 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7710941Z E0605 10:23:38.770000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01182, (ref-fp64): 0.00863 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7715545Z E0605 10:23:38.771000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00877, (ref-fp64): 0.00658 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7720176Z E0605 10:23:38.771000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01019, (ref-fp64): 0.00784 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7725039Z E0605 10:23:38.772000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00974, (ref-fp64): 0.00717 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7729669Z E0605 10:23:38.772000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01295, (ref-fp64): 0.00923 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7733811Z E0605 10:23:38.772000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01250, (ref-fp64): 0.00892 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7738072Z E0605 10:23:38.773000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01303, (ref-fp64): 0.00937 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7742522Z E0605 10:23:38.773000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01262, (ref-fp64): 0.00921 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7746687Z E0605 10:23:38.774000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01255, (ref-fp64): 0.00939 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7751057Z E0605 10:23:38.774000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01175, (ref-fp64): 0.00841 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7755802Z E0605 10:23:38.775000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01002, (ref-fp64): 0.00688 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7760785Z E0605 10:23:38.775000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01099, (ref-fp64): 0.00775 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7765698Z E0605 10:23:38.776000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00999, (ref-fp64): 0.00716 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7770451Z E0605 10:23:38.776000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01167, (ref-fp64): 0.00918 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7774775Z E0605 10:23:38.777000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01131, (ref-fp64): 0.00862 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7778852Z E0605 10:23:38.777000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01217, (ref-fp64): 0.00905 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7783156Z E0605 10:23:38.777000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01197, (ref-fp64): 0.00885 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7787551Z E0605 10:23:38.778000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01229, (ref-fp64): 0.00952 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7791777Z E0605 10:23:38.778000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01280, (ref-fp64): 0.00898 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7796555Z E0605 10:23:38.779000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00907, (ref-fp64): 0.00664 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7800924Z E0605 10:23:38.779000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01051, (ref-fp64): 0.00751 and shape=torch.Size([256, 8, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7805824Z E0605 10:23:38.780000 140039933510272 torch/_dynamo/utils.py:1482] key: layer2.3.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.00707 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7810692Z E0605 10:23:38.780000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01302, (ref-fp64): 0.00903 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7815055Z E0605 10:23:38.781000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01295, (ref-fp64): 0.00913 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7819567Z E0605 10:23:38.781000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01338, (ref-fp64): 0.00943 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7823893Z E0605 10:23:38.781000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01287, (ref-fp64): 0.00842 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7828441Z E0605 10:23:38.782000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01218, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7832920Z E0605 10:23:38.782000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01175, (ref-fp64): 0.00805 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7837793Z E0605 10:23:38.783000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.01009, (ref-fp64): 0.00683 and shape=torch.Size([512, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7842862Z E0605 10:23:38.783000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00961, (ref-fp64): 0.00632 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7847964Z E0605 10:23:38.784000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00943, (ref-fp64): 0.00661 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7852788Z E0605 10:23:38.784000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00902, (ref-fp64): 0.00627 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7857343Z E0605 10:23:38.785000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01218, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7861861Z E0605 10:23:38.785000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01171, (ref-fp64): 0.00799 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7866212Z E0605 10:23:38.786000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01233, (ref-fp64): 0.00860 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7870817Z E0605 10:23:38.786000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01173, (ref-fp64): 0.00805 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7875312Z E0605 10:23:38.787000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01265, (ref-fp64): 0.00923 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7879713Z E0605 10:23:38.787000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01242, (ref-fp64): 0.00850 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7884537Z E0605 10:23:38.788000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01327, (ref-fp64): 0.00906 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7889210Z E0605 10:23:38.788000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01278, (ref-fp64): 0.00870 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7894071Z E0605 10:23:38.788000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00811, (ref-fp64): 0.00609 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7898836Z E0605 10:23:38.789000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.00679 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7903837Z E0605 10:23:38.789000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00979, (ref-fp64): 0.00666 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7908184Z E0605 10:23:38.790000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01188, (ref-fp64): 0.00874 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7912466Z E0605 10:23:38.790000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01181, (ref-fp64): 0.00840 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7916930Z E0605 10:23:38.791000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01223, (ref-fp64): 0.00917 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7920981Z E0605 10:23:38.791000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01190, (ref-fp64): 0.00907 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7925761Z E0605 10:23:38.792000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01343, (ref-fp64): 0.00921 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7930180Z E0605 10:23:38.792000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01317, (ref-fp64): 0.00887 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7934892Z E0605 10:23:38.793000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00886, (ref-fp64): 0.00638 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7939868Z E0605 10:23:38.793000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00986, (ref-fp64): 0.00691 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7945014Z E0605 10:23:38.794000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00984, (ref-fp64): 0.00661 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7949329Z E0605 10:23:38.794000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01312, (ref-fp64): 0.00897 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7953866Z E0605 10:23:38.794000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01328, (ref-fp64): 0.00878 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7957991Z E0605 10:23:38.795000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01273, (ref-fp64): 0.00914 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7962465Z E0605 10:23:38.795000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01241, (ref-fp64): 0.00878 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7967262Z E0605 10:23:38.796000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01357, (ref-fp64): 0.00918 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7971608Z E0605 10:23:38.796000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01315, (ref-fp64): 0.00888 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7976407Z E0605 10:23:38.797000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00952, (ref-fp64): 0.00620 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7981228Z E0605 10:23:38.797000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv2.weight, passes_test: True, RMSE (res-fp64): 0.01013, (ref-fp64): 0.00683 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.7985942Z E0605 10:23:38.798000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.3.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01027, (ref-fp64): 0.00664 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7990262Z E0605 10:23:38.798000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01192, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7994600Z E0605 10:23:38.799000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01191, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.7998832Z E0605 10:23:38.799000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01268, (ref-fp64): 0.00891 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8003277Z E0605 10:23:38.799000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01218, (ref-fp64): 0.00859 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8008067Z E0605 10:23:38.800000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01369, (ref-fp64): 0.00908 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8012406Z E0605 10:23:38.800000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01349, (ref-fp64): 0.00901 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8017044Z E0605 10:23:38.801000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00890, (ref-fp64): 0.00605 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8021958Z E0605 10:23:38.801000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00966, (ref-fp64): 0.00645 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8026626Z E0605 10:23:38.802000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.4.conv3.weight, passes_test: True, RMSE (res-fp64): 0.01005, (ref-fp64): 0.00659 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8030822Z E0605 10:23:38.802000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01258, (ref-fp64): 0.00869 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8035250Z E0605 10:23:38.803000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01221, (ref-fp64): 0.00848 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8039610Z E0605 10:23:38.803000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01205, (ref-fp64): 0.00855 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8044115Z E0605 10:23:38.803000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01158, (ref-fp64): 0.00852 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8048609Z E0605 10:23:38.804000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01363, (ref-fp64): 0.00921 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8053043Z E0605 10:23:38.804000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01331, (ref-fp64): 0.00882 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8057769Z E0605 10:23:38.805000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00888, (ref-fp64): 0.00605 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8062737Z E0605 10:23:38.805000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00639 and shape=torch.Size([512, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8067488Z E0605 10:23:38.806000 140039933510272 torch/_dynamo/utils.py:1482] key: layer3.5.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00950, (ref-fp64): 0.00628 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8071965Z E0605 10:23:38.806000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.00912 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8076541Z E0605 10:23:38.807000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01418, (ref-fp64): 0.00910 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8080950Z E0605 10:23:38.807000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01324, (ref-fp64): 0.00860 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8085696Z E0605 10:23:38.808000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01235, (ref-fp64): 0.00791 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8090149Z E0605 10:23:38.808000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01323, (ref-fp64): 0.00795 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8094598Z E0605 10:23:38.809000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01327, (ref-fp64): 0.00784 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8101398Z E0605 10:23:38.809000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00920, (ref-fp64): 0.00597 and shape=torch.Size([1024, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8105982Z E0605 10:23:38.810000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00901, (ref-fp64): 0.00556 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8117410Z E0605 10:23:38.811000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00900, (ref-fp64): 0.00566 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8129229Z E0605 10:23:38.812000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.0.weight, passes_test: True, RMSE (res-fp64): 0.00838, (ref-fp64): 0.00543 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8133528Z E0605 10:23:38.812000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.bias, passes_test: True, RMSE (res-fp64): 0.01323, (ref-fp64): 0.00795 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8137896Z E0605 10:23:38.813000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.01225, (ref-fp64): 0.00784 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8142255Z E0605 10:23:38.813000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01226, (ref-fp64): 0.00797 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8146645Z E0605 10:23:38.814000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01197, (ref-fp64): 0.00794 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8150883Z E0605 10:23:38.814000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01176, (ref-fp64): 0.00855 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8155469Z E0605 10:23:38.815000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01142, (ref-fp64): 0.00824 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8159838Z E0605 10:23:38.815000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01198, (ref-fp64): 0.00761 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8164450Z E0605 10:23:38.816000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01166, (ref-fp64): 0.00742 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8175860Z E0605 10:23:38.817000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00803, (ref-fp64): 0.00563 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8180486Z E0605 10:23:38.817000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00788, (ref-fp64): 0.00540 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8191996Z E0605 10:23:38.818000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.1.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00760, (ref-fp64): 0.00568 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8196303Z E0605 10:23:38.819000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00990, (ref-fp64): 0.00769 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8200670Z E0605 10:23:38.819000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00981, (ref-fp64): 0.00760 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8205240Z E0605 10:23:38.820000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01370, (ref-fp64): 0.00977 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8209820Z E0605 10:23:38.820000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01179, (ref-fp64): 0.00825 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8214424Z E0605 10:23:38.821000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.00962, (ref-fp64): 0.00568 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8218770Z E0605 10:23:38.821000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00777, (ref-fp64): 0.00791 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8230271Z E0605 10:23:38.822000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00575, (ref-fp64): 0.00432 and shape=torch.Size([1024, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8234956Z E0605 10:23:38.823000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv2.weight, passes_test: True, RMSE (res-fp64): 0.00666, (ref-fp64): 0.00472 and shape=torch.Size([1024, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:23:38.8246324Z E0605 10:23:38.824000 140039933510272 torch/_dynamo/utils.py:1482] key: layer4.2.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00704, (ref-fp64): 0.00514 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:23:38.8570348Z pass
2024-06-05T10:23:38.8685914Z TIMING: entire_frame_compile:55.35279 code_gen:23.83625 inductor_compile:38.98568 backend_compile:46.47559
2024-06-05T10:23:38.8687420Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:14997 | FakeTensorMode.__torch_dispatch__:83610 | ProxyTorchDispatchMode.__torch_dispatch__:18492
2024-06-05T10:23:38.8688612Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique)
2024-06-05T10:23:45.1329484Z 
2024-06-05T10:23:51.1711663Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:23:51.1712252Z loading model: 0it [00:06, ?it/s]
2024-06-05T10:23:51.1714954Z cuda train sam                                
2024-06-05T10:23:51.1720390Z Traceback (most recent call last):
2024-06-05T10:23:51.1721536Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2281, in validate_model
2024-06-05T10:23:51.1722625Z     self.model_iter_fn(model, example_inputs)
2024-06-05T10:23:51.1723570Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 441, in forward_and_backward_pass
2024-06-05T10:23:51.1724480Z     self.grad_scaler.scale(loss).backward()
2024-06-05T10:23:51.1725753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 520, in backward
2024-06-05T10:23:51.1727040Z     torch.autograd.backward(
2024-06-05T10:23:51.1728148Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward
2024-06-05T10:23:51.1728984Z     _engine_run_backward(
2024-06-05T10:23:51.1729947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward
2024-06-05T10:23:51.1731176Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-06-05T10:23:51.1732180Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2024-06-05T10:23:51.1732738Z 
2024-06-05T10:23:51.1733102Z The above exception was the direct cause of the following exception:
2024-06-05T10:23:51.1733574Z 
2024-06-05T10:23:51.1733740Z Traceback (most recent call last):
2024-06-05T10:23:51.1734460Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4139, in run
2024-06-05T10:23:51.1735176Z     ) = runner.load_model(
2024-06-05T10:23:51.1735897Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 363, in load_model
2024-06-05T10:23:51.1736713Z     self.validate_model(model, example_inputs)
2024-06-05T10:23:51.1737534Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2283, in validate_model
2024-06-05T10:23:51.1738363Z     raise RuntimeError("Eager run failed") from e
2024-06-05T10:23:51.1738878Z RuntimeError: Eager run failed
2024-06-05T10:23:51.1739154Z 
2024-06-05T10:23:51.1739287Z eager_fail_to_run
2024-06-05T10:23:54.4805347Z 
2024-06-05T10:23:55.7186385Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:23:55.7187026Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:23:55.7187509Z cuda train shufflenet_v2_x1_0                 
2024-06-05T10:24:21.2866506Z W0605 10:24:21.285000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.2941251Z W0605 10:24:21.293000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.4024028Z W0605 10:24:21.401000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.4061644Z W0605 10:24:21.405000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.5412818Z W0605 10:24:21.540000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.5455882Z W0605 10:24:21.545000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.6037034Z W0605 10:24:21.603000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.6137606Z W0605 10:24:21.613000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.7464524Z W0605 10:24:21.745000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.7492186Z W0605 10:24:21.748000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:21.7690130Z W0605 10:24:21.768000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.4967296Z W0605 10:24:22.496000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.5045118Z W0605 10:24:22.504000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.6147583Z W0605 10:24:22.614000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.6188228Z W0605 10:24:22.618000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.7541592Z W0605 10:24:22.753000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.7583029Z W0605 10:24:22.757000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.8154951Z W0605 10:24:22.814000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.8254472Z W0605 10:24:22.825000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.9607406Z W0605 10:24:22.960000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.9635614Z W0605 10:24:22.963000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:22.9832383Z W0605 10:24:22.982000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.4955667Z W0605 10:24:24.495000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.5033950Z W0605 10:24:24.502000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.6125815Z W0605 10:24:24.612000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.6165623Z W0605 10:24:24.616000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.7611560Z W0605 10:24:24.760000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.7652922Z W0605 10:24:24.764000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.8228034Z W0605 10:24:24.822000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.8329990Z W0605 10:24:24.832000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.9707735Z W0605 10:24:24.970000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.9736789Z W0605 10:24:24.973000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:24.9947734Z W0605 10:24:24.994000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:26.9426585Z W0605 10:24:26.942000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:26.9462599Z W0605 10:24:26.945000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.1217564Z W0605 10:24:27.121000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.1244634Z W0605 10:24:27.124000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.1499647Z W0605 10:24:27.149000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.7282285Z W0605 10:24:27.727000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.7318210Z W0605 10:24:27.731000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.9142406Z W0605 10:24:27.913000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.9172173Z W0605 10:24:27.916000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:27.9426296Z W0605 10:24:27.942000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:29.0370476Z W0605 10:24:29.036000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:29.0409642Z W0605 10:24:29.040000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:29.2313797Z W0605 10:24:29.230000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:29.2340389Z W0605 10:24:29.233000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:29.2619342Z W0605 10:24:29.261000 140071045457664 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:24:33.5396461Z E0605 10:24:33.538000 140076383093376 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01380, (ref-fp64): 0.01241 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5404616Z E0605 10:24:33.539000 140076383093376 torch/_dynamo/utils.py:1482] key: conv1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00054, (ref-fp64): 0.00050 and shape=torch.Size([24, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5410220Z E0605 10:24:33.540000 140076383093376 torch/_dynamo/utils.py:1482] key: conv1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00122, (ref-fp64): 0.00212 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5413795Z E0605 10:24:33.540000 140076383093376 torch/_dynamo/utils.py:1482] key: conv1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00293, (ref-fp64): 0.00241 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5418781Z E0605 10:24:33.541000 140076383093376 torch/_dynamo/utils.py:1482] key: conv5.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00007 and shape=torch.Size([1024, 464, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5423890Z E0605 10:24:33.541000 140076383093376 torch/_dynamo/utils.py:1482] key: conv5.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00044, (ref-fp64): 0.00044 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5428981Z E0605 10:24:33.542000 140076383093376 torch/_dynamo/utils.py:1482] key: conv5.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00056 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5438979Z E0605 10:24:33.543000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00055, (ref-fp64): 0.00077 and shape=torch.Size([24, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5443934Z E0605 10:24:33.543000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00185, (ref-fp64): 0.00272 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5449215Z E0605 10:24:33.544000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00150 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5453813Z E0605 10:24:33.544000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00048, (ref-fp64): 0.00059 and shape=torch.Size([58, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5458056Z E0605 10:24:33.545000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00145, (ref-fp64): 0.00200 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5462648Z E0605 10:24:33.545000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch1.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00192 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5466832Z E0605 10:24:33.546000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00056, (ref-fp64): 0.00082 and shape=torch.Size([58, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5471310Z E0605 10:24:33.546000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00050, (ref-fp64): 0.00067 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5475759Z E0605 10:24:33.547000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00118, (ref-fp64): 0.00128 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5479857Z E0605 10:24:33.547000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00022, (ref-fp64): 0.00035 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5484827Z E0605 10:24:33.548000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00083, (ref-fp64): 0.00082 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5489699Z E0605 10:24:33.548000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00098, (ref-fp64): 0.00112 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5494584Z E0605 10:24:33.549000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00033, (ref-fp64): 0.00034 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5499102Z E0605 10:24:33.549000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00143, (ref-fp64): 0.00143 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5503741Z E0605 10:24:33.549000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.0.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00124, (ref-fp64): 0.00127 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5508827Z E0605 10:24:33.550000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00030, (ref-fp64): 0.00035 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5513484Z E0605 10:24:33.550000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00114 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5518337Z E0605 10:24:33.551000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00118, (ref-fp64): 0.00157 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5523354Z E0605 10:24:33.551000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00029, (ref-fp64): 0.00027 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5528412Z E0605 10:24:33.552000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00073 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5533060Z E0605 10:24:33.552000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00085, (ref-fp64): 0.00112 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5538070Z E0605 10:24:33.553000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00031, (ref-fp64): 0.00031 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5542782Z E0605 10:24:33.553000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00143, (ref-fp64): 0.00139 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5547822Z E0605 10:24:33.554000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.1.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00125, (ref-fp64): 0.00119 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5553249Z E0605 10:24:33.554000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00056, (ref-fp64): 0.00061 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5557562Z E0605 10:24:33.555000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00164, (ref-fp64): 0.00214 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5562436Z E0605 10:24:33.555000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00135, (ref-fp64): 0.00093 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5569760Z E0605 10:24:33.556000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00167 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5574869Z E0605 10:24:33.557000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00073, (ref-fp64): 0.00046 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5582236Z E0605 10:24:33.557000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00183, (ref-fp64): 0.00159 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5586885Z E0605 10:24:33.558000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.2.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00078, (ref-fp64): 0.00064 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5591863Z E0605 10:24:33.558000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00043, (ref-fp64): 0.00045 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5596669Z E0605 10:24:33.559000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00225, (ref-fp64): 0.00200 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5601575Z E0605 10:24:33.559000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00152, (ref-fp64): 0.00182 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5607044Z E0605 10:24:33.560000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00049, (ref-fp64): 0.00034 and shape=torch.Size([58, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5611891Z E0605 10:24:33.560000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00123, (ref-fp64): 0.00133 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5616607Z E0605 10:24:33.561000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00085 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5621566Z E0605 10:24:33.561000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00028, (ref-fp64): 0.00028 and shape=torch.Size([58, 58, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5626453Z E0605 10:24:33.562000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00173, (ref-fp64): 0.00202 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5631053Z E0605 10:24:33.562000 140076383093376 torch/_dynamo/utils.py:1482] key: stage2.3.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00122, (ref-fp64): 0.00157 and shape=torch.Size([58]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5637987Z E0605 10:24:33.563000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00079, (ref-fp64): 0.00089 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5642893Z E0605 10:24:33.563000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00070, (ref-fp64): 0.00089 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5648211Z E0605 10:24:33.564000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch1.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00027 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5653065Z E0605 10:24:33.564000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch1.3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00096 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5657961Z E0605 10:24:33.565000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch1.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00119, (ref-fp64): 0.00103 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5663028Z E0605 10:24:33.565000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00050, (ref-fp64): 0.00038 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5667932Z E0605 10:24:33.566000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00166, (ref-fp64): 0.00127 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5672779Z E0605 10:24:33.566000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00195, (ref-fp64): 0.00147 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5679913Z E0605 10:24:33.567000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00065 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5685003Z E0605 10:24:33.568000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00095, (ref-fp64): 0.00062 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5690255Z E0605 10:24:33.568000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00023 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5695185Z E0605 10:24:33.569000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00103 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5700110Z E0605 10:24:33.569000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.0.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00077 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5704918Z E0605 10:24:33.570000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00021 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5709319Z E0605 10:24:33.570000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00059, (ref-fp64): 0.00060 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5713637Z E0605 10:24:33.570000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00098 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5718162Z E0605 10:24:33.571000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00032, (ref-fp64): 0.00028 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5722649Z E0605 10:24:33.571000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00087, (ref-fp64): 0.00078 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5727509Z E0605 10:24:33.572000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00067 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5732229Z E0605 10:24:33.572000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00024 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5736682Z E0605 10:24:33.573000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00106, (ref-fp64): 0.00123 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5741032Z E0605 10:24:33.573000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.1.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00061 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5745576Z E0605 10:24:33.574000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00029, (ref-fp64): 0.00038 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5749962Z E0605 10:24:33.574000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00156, (ref-fp64): 0.00215 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5754387Z E0605 10:24:33.575000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00162, (ref-fp64): 0.00204 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5759069Z E0605 10:24:33.575000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00028, (ref-fp64): 0.00036 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5763530Z E0605 10:24:33.575000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00112 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5768087Z E0605 10:24:33.576000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00111 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5772641Z E0605 10:24:33.576000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00029, (ref-fp64): 0.00030 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5777101Z E0605 10:24:33.577000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00124, (ref-fp64): 0.00133 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5781490Z E0605 10:24:33.577000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.2.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00153, (ref-fp64): 0.00166 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5786078Z E0605 10:24:33.578000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00041, (ref-fp64): 0.00044 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5790382Z E0605 10:24:33.578000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00157, (ref-fp64): 0.00165 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5794811Z E0605 10:24:33.579000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00153, (ref-fp64): 0.00156 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5801756Z E0605 10:24:33.579000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00111, (ref-fp64): 0.00097 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5806307Z E0605 10:24:33.580000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00087, (ref-fp64): 0.00084 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5811198Z E0605 10:24:33.580000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00022 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5815536Z E0605 10:24:33.581000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00149, (ref-fp64): 0.00138 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5819855Z E0605 10:24:33.581000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.3.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00091, (ref-fp64): 0.00078 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5824480Z E0605 10:24:33.581000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00035, (ref-fp64): 0.00027 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5828837Z E0605 10:24:33.582000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00160, (ref-fp64): 0.00120 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5833316Z E0605 10:24:33.582000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00135, (ref-fp64): 0.00137 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5837958Z E0605 10:24:33.583000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00023, (ref-fp64): 0.00022 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5842274Z E0605 10:24:33.583000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00109, (ref-fp64): 0.00105 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5847235Z E0605 10:24:33.584000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00081, (ref-fp64): 0.00082 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5852138Z E0605 10:24:33.584000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00019 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5856560Z E0605 10:24:33.585000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00160, (ref-fp64): 0.00135 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5861301Z E0605 10:24:33.585000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.4.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00110, (ref-fp64): 0.00093 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5866782Z E0605 10:24:33.586000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00044, (ref-fp64): 0.00035 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5870957Z E0605 10:24:33.586000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00189, (ref-fp64): 0.00148 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5883560Z E0605 10:24:33.587000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00129, (ref-fp64): 0.00101 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5886735Z E0605 10:24:33.587000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00175, (ref-fp64): 0.00133 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5889679Z E0605 10:24:33.588000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00087, (ref-fp64): 0.00069 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5892496Z E0605 10:24:33.588000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00021 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5896905Z E0605 10:24:33.589000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00217, (ref-fp64): 0.00188 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5901499Z E0605 10:24:33.589000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.5.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00156, (ref-fp64): 0.00137 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5906382Z E0605 10:24:33.590000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00035 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5910967Z E0605 10:24:33.590000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00128, (ref-fp64): 0.00124 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5915620Z E0605 10:24:33.591000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00109, (ref-fp64): 0.00104 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5922321Z E0605 10:24:33.591000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00084, (ref-fp64): 0.00097 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5927233Z E0605 10:24:33.592000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00076, (ref-fp64): 0.00079 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5932046Z E0605 10:24:33.592000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00023, (ref-fp64): 0.00026 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5936569Z E0605 10:24:33.593000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00111, (ref-fp64): 0.00129 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5940983Z E0605 10:24:33.593000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.6.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00113, (ref-fp64): 0.00118 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5945834Z E0605 10:24:33.594000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00039, (ref-fp64): 0.00043 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5950322Z E0605 10:24:33.594000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00157, (ref-fp64): 0.00153 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5955147Z E0605 10:24:33.595000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00109, (ref-fp64): 0.00136 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5959902Z E0605 10:24:33.595000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00020 and shape=torch.Size([116, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.5964780Z E0605 10:24:33.596000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00091, (ref-fp64): 0.00112 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5969275Z E0605 10:24:33.596000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00077, (ref-fp64): 0.00076 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5974133Z E0605 10:24:33.596000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00023 and shape=torch.Size([116, 116, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5978634Z E0605 10:24:33.597000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00141, (ref-fp64): 0.00189 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5983213Z E0605 10:24:33.597000 140076383093376 torch/_dynamo/utils.py:1482] key: stage3.7.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00125, (ref-fp64): 0.00148 and shape=torch.Size([116]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5989785Z E0605 10:24:33.598000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00088 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5994466Z E0605 10:24:33.599000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00100 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.5999380Z E0605 10:24:33.599000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch1.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00026 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6003938Z E0605 10:24:33.599000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch1.3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00111 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6008954Z E0605 10:24:33.600000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch1.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00125, (ref-fp64): 0.00128 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6014126Z E0605 10:24:33.600000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00026 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6018893Z E0605 10:24:33.601000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00075, (ref-fp64): 0.00096 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6023398Z E0605 10:24:33.601000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00135 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6030165Z E0605 10:24:33.602000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00065, (ref-fp64): 0.00063 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6034717Z E0605 10:24:33.603000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00045, (ref-fp64): 0.00050 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6039514Z E0605 10:24:33.603000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00014, (ref-fp64): 0.00015 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6044455Z E0605 10:24:33.604000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00088, (ref-fp64): 0.00091 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6049443Z E0605 10:24:33.604000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.0.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00061, (ref-fp64): 0.00063 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6054410Z E0605 10:24:33.605000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00027, (ref-fp64): 0.00030 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6058749Z E0605 10:24:33.605000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00123, (ref-fp64): 0.00138 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6063085Z E0605 10:24:33.605000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00149, (ref-fp64): 0.00137 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6068128Z E0605 10:24:33.606000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00016 and shape=torch.Size([232, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:33.6072437Z E0605 10:24:33.606000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00080 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6077069Z E0605 10:24:33.607000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00081 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6082364Z E0605 10:24:33.607000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00017 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6087506Z E0605 10:24:33.608000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00106, (ref-fp64): 0.00090 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6092573Z E0605 10:24:33.608000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.1.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00072, (ref-fp64): 0.00067 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6097654Z E0605 10:24:33.609000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00018, (ref-fp64): 0.00017 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6102899Z E0605 10:24:33.609000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00103 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6107697Z E0605 10:24:33.610000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00095 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6114849Z E0605 10:24:33.611000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00052, (ref-fp64): 0.00050 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6119672Z E0605 10:24:33.611000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00055, (ref-fp64): 0.00050 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6125253Z E0605 10:24:33.612000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00012, (ref-fp64): 0.00011 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6130692Z E0605 10:24:33.612000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00056, (ref-fp64): 0.00053 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6135899Z E0605 10:24:33.613000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.2.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00050, (ref-fp64): 0.00048 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6141193Z E0605 10:24:33.613000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00019, (ref-fp64): 0.00019 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6146430Z E0605 10:24:33.614000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00093 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6151199Z E0605 10:24:33.614000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00102, (ref-fp64): 0.00108 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6158133Z E0605 10:24:33.615000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.00044, (ref-fp64): 0.00039 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6163279Z E0605 10:24:33.615000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.00047, (ref-fp64): 0.00041 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6168917Z E0605 10:24:33.616000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.5.weight.grad, passes_test: True, RMSE (res-fp64): 0.00012, (ref-fp64): 0.00010 and shape=torch.Size([232, 232, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6173253Z E0605 10:24:33.616000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00047, (ref-fp64): 0.00041 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6178081Z E0605 10:24:33.617000 140076383093376 torch/_dynamo/utils.py:1482] key: stage4.3.branch2.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00033, (ref-fp64): 0.00033 and shape=torch.Size([232]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:24:33.6877214Z pass
2024-06-05T10:24:33.6888317Z TIMING: entire_frame_compile:18.32719 code_gen:15.76328 inductor_compile:24.27312 backend_compile:16.38774
2024-06-05T10:24:33.6889940Z STATS: call_* op count: 275 | FakeTensor.__torch_dispatch__:7885 | FakeTensorMode.__torch_dispatch__:46847 | ProxyTorchDispatchMode.__torch_dispatch__:12199
2024-06-05T10:24:33.6891344Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique)
2024-06-05T10:24:38.3436606Z 
2024-06-05T10:24:38.8402497Z loading model: 0it [00:00, ?it/s]/opt/conda/envs/py_3.10/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
2024-06-05T10:24:38.8404033Z   if not isinstance(terminated, (bool, np.bool8)):
2024-06-05T10:24:39.1370162Z 
2024-06-05T10:24:39.1370797Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:24:39.1371420Z cuda train soft_actor_critic                  
2024-06-05T10:24:45.3566322Z W0605 10:24:45.355000 139673896669824 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:24:48.4559743Z E0605 10:24:48.455000 139673896669824 torch/_dynamo/utils.py:1482] key: fc3.weight.grad, passes_test: True, RMSE (res-fp64): 0.01210, (ref-fp64): 0.01151 and shape=torch.Size([2, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:24:48.4575779Z pass
2024-06-05T10:24:48.4584758Z TIMING: entire_frame_compile:5.88841 code_gen:3.67611 inductor_compile:4.28288 backend_compile:5.3668
2024-06-05T10:24:48.4586075Z STATS: call_* op count: 57 | FakeTensor.__torch_dispatch__:598 | FakeTensorMode.__torch_dispatch__:3864 | ProxyTorchDispatchMode.__torch_dispatch__:894
2024-06-05T10:24:48.4587244Z Dynamo produced 4 graphs covering 57 ops with 6 graph breaks (5 unique)
2024-06-05T10:24:52.0666573Z 
2024-06-05T10:24:53.3112300Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:24:53.3112830Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:24:53.3113320Z cuda train speech_transformer                 
2024-06-05T10:25:08.8873225Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_104). Found from : 
2024-06-05T10:25:08.8874906Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 58, in torch_dynamo_resume_in_forward_at_50
2024-06-05T10:25:08.8876419Z     enc_output, enc_slf_attn = enc_layer(
2024-06-05T10:25:08.8877551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:08.8878465Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:08.8879649Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 84, in forward
2024-06-05T10:25:08.8880826Z     enc_output, enc_slf_attn = self.slf_attn(
2024-06-05T10:25:08.8881971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:08.8883105Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:08.8884276Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-06-05T10:25:08.8885482Z     output, attn = self.attention(q, k, v, mask=mask)
2024-06-05T10:25:08.8886780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:08.8887660Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:08.8888814Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-06-05T10:25:08.8889956Z     attn = attn / self.temperature
2024-06-05T10:25:08.8890251Z 
2024-06-05T10:25:30.6399874Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_163). Found from : 
2024-06-05T10:25:30.6401599Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 105, in torch_dynamo_resume_in_forward_at_96
2024-06-05T10:25:30.6403084Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2024-06-05T10:25:30.6404357Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:30.6405558Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:30.6407089Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 243, in forward
2024-06-05T10:25:30.6408284Z     dec_output, dec_slf_attn = self.slf_attn(
2024-06-05T10:25:30.6409345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:30.6410278Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:30.6411524Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-06-05T10:25:30.6412750Z     output, attn = self.attention(q, k, v, mask=mask)
2024-06-05T10:25:30.6413817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:25:30.6414733Z     return forward_call(*args, **kwargs)
2024-06-05T10:25:30.6415913Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-06-05T10:25:30.6417075Z     attn = attn / self.temperature
2024-06-05T10:25:30.6417389Z 
2024-06-05T10:25:48.4337004Z E0605 10:25:48.432000 140125425275520 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00255, (ref-fp64): 0.00254 and shape=torch.Size([10, 22, 1014]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:25:48.5430522Z pass
2024-06-05T10:25:48.5790180Z TIMING: entire_frame_compile:34.21032 inductor_compile:29.50027 backend_compile:31.67327 code_gen:18.9257
2024-06-05T10:25:48.5791587Z STATS: call_* op count: 771 | FakeTensorMode.__torch_dispatch__:71949 | ProxyTorchDispatchMode.__torch_dispatch__:25322 | FakeTensor.__torch_dispatch__:7854
2024-06-05T10:25:48.5792807Z Dynamo produced 10 graphs covering 771 ops with 16 graph breaks (7 unique)
2024-06-05T10:25:53.7522972Z 
2024-06-05T10:25:54.9045553Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:25:54.9046055Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:25:54.9046683Z cuda train squeezenet1_1                      
2024-06-05T10:26:10.8867190Z E0605 10:26:10.885000 140326616830592 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.01934, (ref-fp64): 0.02506 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8872020Z E0605 10:26:10.886000 140326616830592 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00724, (ref-fp64): 0.01045 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8879754Z E0605 10:26:10.887000 140326616830592 torch/_dynamo/utils.py:1482] key: classifier.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00003 and shape=torch.Size([1000, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8886268Z E0605 10:26:10.888000 140326616830592 torch/_dynamo/utils.py:1482] key: features.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00028, (ref-fp64): 0.00025 and shape=torch.Size([64, 3, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8893371Z E0605 10:26:10.888000 140326616830592 torch/_dynamo/utils.py:1482] key: features.10.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00131, (ref-fp64): 0.00151 and shape=torch.Size([192, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8900469Z E0605 10:26:10.889000 140326616830592 torch/_dynamo/utils.py:1482] key: features.10.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00083, (ref-fp64): 0.00119 and shape=torch.Size([192, 48, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8907617Z E0605 10:26:10.890000 140326616830592 torch/_dynamo/utils.py:1482] key: features.10.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00098, (ref-fp64): 0.00121 and shape=torch.Size([48, 384, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8913340Z E0605 10:26:10.890000 140326616830592 torch/_dynamo/utils.py:1482] key: features.11.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00123 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8919672Z E0605 10:26:10.891000 140326616830592 torch/_dynamo/utils.py:1482] key: features.11.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00080 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8926286Z E0605 10:26:10.892000 140326616830592 torch/_dynamo/utils.py:1482] key: features.11.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00116, (ref-fp64): 0.00163 and shape=torch.Size([64, 384, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8932883Z E0605 10:26:10.892000 140326616830592 torch/_dynamo/utils.py:1482] key: features.12.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00131 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8939317Z E0605 10:26:10.893000 140326616830592 torch/_dynamo/utils.py:1482] key: features.12.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00029 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8945931Z E0605 10:26:10.894000 140326616830592 torch/_dynamo/utils.py:1482] key: features.12.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00123 and shape=torch.Size([64, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8952082Z E0605 10:26:10.894000 140326616830592 torch/_dynamo/utils.py:1482] key: features.3.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00069, (ref-fp64): 0.00061 and shape=torch.Size([64, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8958379Z E0605 10:26:10.895000 140326616830592 torch/_dynamo/utils.py:1482] key: features.3.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00065, (ref-fp64): 0.00042 and shape=torch.Size([64, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8964877Z E0605 10:26:10.896000 140326616830592 torch/_dynamo/utils.py:1482] key: features.3.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00062, (ref-fp64): 0.00078 and shape=torch.Size([16, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8971391Z E0605 10:26:10.896000 140326616830592 torch/_dynamo/utils.py:1482] key: features.4.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00091, (ref-fp64): 0.00097 and shape=torch.Size([64, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8977408Z E0605 10:26:10.897000 140326616830592 torch/_dynamo/utils.py:1482] key: features.4.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00054, (ref-fp64): 0.00062 and shape=torch.Size([64, 16, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.8983917Z E0605 10:26:10.897000 140326616830592 torch/_dynamo/utils.py:1482] key: features.4.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00058, (ref-fp64): 0.00047 and shape=torch.Size([16, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8989949Z E0605 10:26:10.898000 140326616830592 torch/_dynamo/utils.py:1482] key: features.6.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00069, (ref-fp64): 0.00074 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.8996567Z E0605 10:26:10.899000 140326616830592 torch/_dynamo/utils.py:1482] key: features.6.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00049, (ref-fp64): 0.00061 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.9002896Z E0605 10:26:10.899000 140326616830592 torch/_dynamo/utils.py:1482] key: features.6.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00067, (ref-fp64): 0.00081 and shape=torch.Size([32, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.9009878Z E0605 10:26:10.900000 140326616830592 torch/_dynamo/utils.py:1482] key: features.7.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00101, (ref-fp64): 0.00108 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.9016324Z E0605 10:26:10.901000 140326616830592 torch/_dynamo/utils.py:1482] key: features.7.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00077, (ref-fp64): 0.00087 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.9022876Z E0605 10:26:10.901000 140326616830592 torch/_dynamo/utils.py:1482] key: features.7.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00071, (ref-fp64): 0.00083 and shape=torch.Size([32, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.9028826Z E0605 10:26:10.902000 140326616830592 torch/_dynamo/utils.py:1482] key: features.9.expand1x1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00083 and shape=torch.Size([192, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.9035926Z E0605 10:26:10.903000 140326616830592 torch/_dynamo/utils.py:1482] key: features.9.expand3x3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00075, (ref-fp64): 0.00090 and shape=torch.Size([192, 48, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:26:10.9042197Z E0605 10:26:10.903000 140326616830592 torch/_dynamo/utils.py:1482] key: features.9.squeeze.weight.grad, passes_test: True, RMSE (res-fp64): 0.00088, (ref-fp64): 0.00113 and shape=torch.Size([48, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:26:10.9149235Z pass
2024-06-05T10:26:10.9171709Z TIMING: entire_frame_compile:5.4982 code_gen:9.24384 inductor_compile:11.04082 backend_compile:4.7823
2024-06-05T10:26:10.9173106Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:1308 | FakeTensorMode.__torch_dispatch__:8848 | ProxyTorchDispatchMode.__torch_dispatch__:2530
2024-06-05T10:26:10.9174478Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique)
2024-06-05T10:26:14.8307894Z 
2024-06-05T10:26:15.8984274Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:26:15.8984697Z 
2024-06-05T10:26:16.0111403Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-06-05T10:26:16.0112262Z 
2024-06-05T10:26:17.0428480Z Loading pipeline components...:  33% 2/6 [00:00<00:00, 17.80it/s]
2024-06-05T10:26:17.0429597Z 
2024-06-05T10:26:17.0434149Z Loading pipeline components...:  67% 4/6 [00:01<00:00,  3.06it/s]
2024-06-05T10:26:17.0435130Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.24it/s]
2024-06-05T10:26:18.3414020Z 
2024-06-05T10:26:18.3414645Z loading model: 0it [00:03, ?it/s]
2024-06-05T10:26:18.3415301Z cuda train stable_diffusion_text_encoder      
2024-06-05T10:26:18.3782322Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2024-06-05T10:27:12.9723376Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:27:12.9724641Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 439, in torch_dynamo_resume_in_forward_and_backward_pass_at_434
2024-06-05T10:27:12.9725659Z     pred = mod(*cloned_inputs)
2024-06-05T10:27:12.9726868Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:27:12.9727769Z     return forward_call(*args, **kwargs)
2024-06-05T10:27:12.9728821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 806, in forward
2024-06-05T10:27:12.9729748Z     return self.text_model(
2024-06-05T10:27:12.9730670Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:27:12.9731642Z     return forward_call(*args, **kwargs)
2024-06-05T10:27:12.9732797Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 699, in forward
2024-06-05T10:27:12.9733980Z     hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
2024-06-05T10:27:12.9735151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:27:12.9736052Z     return forward_call(*args, **kwargs)
2024-06-05T10:27:12.9737107Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 224, in forward
2024-06-05T10:27:12.9738107Z     inputs_embeds = self.token_embedding(input_ids)
2024-06-05T10:27:12.9738474Z 
2024-06-05T10:27:14.2029374Z pass
2024-06-05T10:27:14.2532916Z TIMING: entire_frame_compile:37.82445 code_gen:15.23528 inductor_compile:28.79513 backend_compile:33.60871
2024-06-05T10:27:14.2534316Z STATS: call_* op count: 828 | FakeTensorMode.__torch_dispatch__:99140 | FakeTensor.__torch_dispatch__:13646 | ProxyTorchDispatchMode.__torch_dispatch__:32995
2024-06-05T10:27:14.2535531Z Dynamo produced 2 graphs covering 828 ops with 5 graph breaks (5 unique)
2024-06-05T10:27:19.7392737Z 
2024-06-05T10:27:20.8251775Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:27:20.8252148Z 
2024-06-05T10:27:21.5811329Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-06-05T10:27:21.5811821Z 
2024-06-05T10:27:21.9289122Z Loading pipeline components...:  17% 1/6 [00:00<00:03,  1.32it/s]
2024-06-05T10:27:21.9290121Z 
2024-06-05T10:27:21.9753131Z Loading pipeline components...:  83% 5/6 [00:01<00:00,  5.36it/s]
2024-06-05T10:27:21.9753945Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.22it/s]
2024-06-05T10:27:23.8241502Z 
2024-06-05T10:27:23.8242892Z loading model: 0it [00:04, ?it/s]
2024-06-05T10:27:23.8243719Z cuda train stable_diffusion_unet              
2024-06-05T10:27:23.8247821Z pass_due_to_skip
2024-06-05T10:27:24.1852869Z TIMING:
2024-06-05T10:27:24.1853231Z STATS: call_* op count: 0
2024-06-05T10:27:24.1854026Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-06-05T10:27:27.5592318Z 
2024-06-05T10:27:30.6369650Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:27:30.6370149Z loading model: 0it [00:03, ?it/s]
2024-06-05T10:27:30.6370636Z cuda train timm_efficientnet                  
2024-06-05T10:28:14.6793458Z W0605 10:28:14.678000 140570087031424 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:29:13.1866555Z E0605 10:29:13.185000 140570087031424 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00566, (ref-fp64): 0.00569 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2302411Z E0605 10:29:13.229000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01461, (ref-fp64): 0.01856 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2305526Z E0605 10:29:13.230000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00943, (ref-fp64): 0.01602 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2309815Z E0605 10:29:13.230000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01552, (ref-fp64): 0.02334 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2314316Z E0605 10:29:13.230000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00894, (ref-fp64): 0.01568 and shape=torch.Size([16]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2318932Z E0605 10:29:13.231000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00983, (ref-fp64): 0.01409 and shape=torch.Size([32, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2323647Z E0605 10:29:13.231000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.01009, (ref-fp64): 0.01638 and shape=torch.Size([16, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2328460Z E0605 10:29:13.232000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00908, (ref-fp64): 0.01267 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2332820Z E0605 10:29:13.232000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00479, (ref-fp64): 0.00490 and shape=torch.Size([32, 8, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2337176Z E0605 10:29:13.233000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.0.0.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00937, (ref-fp64): 0.01266 and shape=torch.Size([8]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2344079Z E0605 10:29:13.233000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01442, (ref-fp64): 0.01863 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2348333Z E0605 10:29:13.234000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00910, (ref-fp64): 0.00835 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2352514Z E0605 10:29:13.234000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01881, (ref-fp64): 0.01682 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2356995Z E0605 10:29:13.235000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01176, (ref-fp64): 0.00829 and shape=torch.Size([96]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2361460Z E0605 10:29:13.235000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01795, (ref-fp64): 0.01119 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2366111Z E0605 10:29:13.236000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01167, (ref-fp64): 0.00981 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2371020Z E0605 10:29:13.236000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00813, (ref-fp64): 0.01198 and shape=torch.Size([96, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2375386Z E0605 10:29:13.237000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00923, (ref-fp64): 0.01541 and shape=torch.Size([96, 16, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2379787Z E0605 10:29:13.237000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00797, (ref-fp64): 0.01143 and shape=torch.Size([24, 96, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2391828Z E0605 10:29:13.238000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01348, (ref-fp64): 0.01687 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2396336Z E0605 10:29:13.239000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00706, (ref-fp64): 0.00699 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2400523Z E0605 10:29:13.239000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01716, (ref-fp64): 0.01532 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2404964Z E0605 10:29:13.240000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00787, (ref-fp64): 0.00709 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2409587Z E0605 10:29:13.240000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01880, (ref-fp64): 0.01604 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2413787Z E0605 10:29:13.240000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01060, (ref-fp64): 0.00782 and shape=torch.Size([24]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2418265Z E0605 10:29:13.241000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00720, (ref-fp64): 0.00821 and shape=torch.Size([144, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2422670Z E0605 10:29:13.241000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00758, (ref-fp64): 0.01118 and shape=torch.Size([144, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2427078Z E0605 10:29:13.242000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.1.1.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00709, (ref-fp64): 0.00721 and shape=torch.Size([24, 144, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2439255Z E0605 10:29:13.243000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01527, (ref-fp64): 0.01604 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2443833Z E0605 10:29:13.243000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.00892 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2448484Z E0605 10:29:13.244000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01184, (ref-fp64): 0.01248 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2452739Z E0605 10:29:13.244000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.01110, (ref-fp64): 0.00884 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2456923Z E0605 10:29:13.245000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01190, (ref-fp64): 0.01268 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2461086Z E0605 10:29:13.245000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01075, (ref-fp64): 0.01014 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2465669Z E0605 10:29:13.246000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00815, (ref-fp64): 0.00936 and shape=torch.Size([144, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2470105Z E0605 10:29:13.246000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00984, (ref-fp64): 0.01266 and shape=torch.Size([144, 24, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2474932Z E0605 10:29:13.247000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00806, (ref-fp64): 0.00832 and shape=torch.Size([40, 144, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2479181Z E0605 10:29:13.247000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00811, (ref-fp64): 0.00809 and shape=torch.Size([144]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2490096Z E0605 10:29:13.248000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01525, (ref-fp64): 0.01569 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2494606Z E0605 10:29:13.249000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00818, (ref-fp64): 0.00703 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2499026Z E0605 10:29:13.249000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01100, (ref-fp64): 0.01343 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2503412Z E0605 10:29:13.249000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00738, (ref-fp64): 0.00698 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2507867Z E0605 10:29:13.250000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01278, (ref-fp64): 0.01348 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2512187Z E0605 10:29:13.250000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00978, (ref-fp64): 0.00829 and shape=torch.Size([40]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2516955Z E0605 10:29:13.251000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00690, (ref-fp64): 0.00691 and shape=torch.Size([240, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2521473Z E0605 10:29:13.251000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00745, (ref-fp64): 0.00773 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2526358Z E0605 10:29:13.252000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00722, (ref-fp64): 0.00744 and shape=torch.Size([40, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2530760Z E0605 10:29:13.252000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.2.1.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00697, (ref-fp64): 0.00709 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2540999Z E0605 10:29:13.253000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01362, (ref-fp64): 0.01466 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2545333Z E0605 10:29:13.254000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00948, (ref-fp64): 0.00805 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2549642Z E0605 10:29:13.254000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01006, (ref-fp64): 0.01012 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2554229Z E0605 10:29:13.254000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00833, (ref-fp64): 0.00810 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2558778Z E0605 10:29:13.255000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01124, (ref-fp64): 0.01113 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2562983Z E0605 10:29:13.255000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00904, (ref-fp64): 0.00949 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2567865Z E0605 10:29:13.256000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00815, (ref-fp64): 0.00823 and shape=torch.Size([240, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2572386Z E0605 10:29:13.256000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00875, (ref-fp64): 0.00945 and shape=torch.Size([240, 40, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2576866Z E0605 10:29:13.257000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00795, (ref-fp64): 0.00811 and shape=torch.Size([80, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2581339Z E0605 10:29:13.257000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00742, (ref-fp64): 0.00781 and shape=torch.Size([240]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2585697Z E0605 10:29:13.258000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00581, (ref-fp64): 0.00584 and shape=torch.Size([240, 10, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2590019Z E0605 10:29:13.258000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.0.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00517, (ref-fp64): 0.00787 and shape=torch.Size([10]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2596373Z E0605 10:29:13.259000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01088, (ref-fp64): 0.01274 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2600913Z E0605 10:29:13.259000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00770, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2605573Z E0605 10:29:13.260000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01021, (ref-fp64): 0.01063 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2609931Z E0605 10:29:13.260000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00743, (ref-fp64): 0.00713 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2614374Z E0605 10:29:13.261000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01016, (ref-fp64): 0.01104 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2619065Z E0605 10:29:13.261000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00785, (ref-fp64): 0.00781 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2623683Z E0605 10:29:13.261000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00726, (ref-fp64): 0.00766 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2628197Z E0605 10:29:13.262000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00760, (ref-fp64): 0.00773 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2632930Z E0605 10:29:13.262000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00719, (ref-fp64): 0.00754 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2637299Z E0605 10:29:13.263000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00714, (ref-fp64): 0.00721 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2643891Z E0605 10:29:13.263000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.1.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00729, (ref-fp64): 0.00733 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2650372Z E0605 10:29:13.264000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01075, (ref-fp64): 0.01278 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2654762Z E0605 10:29:13.265000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00752, (ref-fp64): 0.00720 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2659196Z E0605 10:29:13.265000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01012 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2663428Z E0605 10:29:13.265000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00745, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2667725Z E0605 10:29:13.266000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01089, (ref-fp64): 0.01065 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2672006Z E0605 10:29:13.266000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00915, (ref-fp64): 0.00803 and shape=torch.Size([80]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2676562Z E0605 10:29:13.267000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00724, (ref-fp64): 0.00757 and shape=torch.Size([480, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2681113Z E0605 10:29:13.267000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00776, (ref-fp64): 0.00811 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2685875Z E0605 10:29:13.268000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00779, (ref-fp64): 0.00790 and shape=torch.Size([80, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2690439Z E0605 10:29:13.268000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00715, (ref-fp64): 0.00717 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2694995Z E0605 10:29:13.269000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00555, (ref-fp64): 0.00556 and shape=torch.Size([480, 20, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2699371Z E0605 10:29:13.269000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.3.2.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00651, (ref-fp64): 0.00736 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2705921Z E0605 10:29:13.270000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01086, (ref-fp64): 0.01108 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2710359Z E0605 10:29:13.270000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00890, (ref-fp64): 0.00790 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2714880Z E0605 10:29:13.271000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01081, (ref-fp64): 0.01091 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2719228Z E0605 10:29:13.271000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00875, (ref-fp64): 0.00791 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2723827Z E0605 10:29:13.271000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01053, (ref-fp64): 0.01044 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2728831Z E0605 10:29:13.272000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00899, (ref-fp64): 0.00932 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2733196Z E0605 10:29:13.272000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00885, (ref-fp64): 0.00906 and shape=torch.Size([480, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2737724Z E0605 10:29:13.273000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00897, (ref-fp64): 0.00929 and shape=torch.Size([480, 80, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2742494Z E0605 10:29:13.273000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00850, (ref-fp64): 0.00874 and shape=torch.Size([112, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2747007Z E0605 10:29:13.274000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00829, (ref-fp64): 0.00763 and shape=torch.Size([480]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2751681Z E0605 10:29:13.274000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00664, (ref-fp64): 0.00657 and shape=torch.Size([480, 20, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2755997Z E0605 10:29:13.275000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00782, (ref-fp64): 0.00928 and shape=torch.Size([20]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2760623Z E0605 10:29:13.275000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.0.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00606, (ref-fp64): 0.00607 and shape=torch.Size([20, 480, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2765335Z E0605 10:29:13.276000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01074, (ref-fp64): 0.01079 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2769672Z E0605 10:29:13.276000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00859, (ref-fp64): 0.00736 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2774086Z E0605 10:29:13.277000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01069, (ref-fp64): 0.01073 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2778687Z E0605 10:29:13.277000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00857, (ref-fp64): 0.00735 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2783096Z E0605 10:29:13.277000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01066, (ref-fp64): 0.01064 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2787534Z E0605 10:29:13.278000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00885, (ref-fp64): 0.00771 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2792262Z E0605 10:29:13.278000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00846, (ref-fp64): 0.00857 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2797137Z E0605 10:29:13.279000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00861, (ref-fp64): 0.00861 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2802041Z E0605 10:29:13.279000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00816, (ref-fp64): 0.00831 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2807080Z E0605 10:29:13.280000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00766, (ref-fp64): 0.00733 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2811670Z E0605 10:29:13.280000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00582, (ref-fp64): 0.00583 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2815802Z E0605 10:29:13.281000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00829, (ref-fp64): 0.00837 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2820352Z E0605 10:29:13.281000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.1.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00519, (ref-fp64): 0.00519 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2824713Z E0605 10:29:13.282000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01051, (ref-fp64): 0.01137 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2828885Z E0605 10:29:13.282000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00832, (ref-fp64): 0.00746 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2833526Z E0605 10:29:13.282000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01019, (ref-fp64): 0.01046 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2837781Z E0605 10:29:13.283000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00829, (ref-fp64): 0.00746 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2842640Z E0605 10:29:13.283000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01078, (ref-fp64): 0.01080 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2847154Z E0605 10:29:13.284000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00882, (ref-fp64): 0.00781 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2852032Z E0605 10:29:13.284000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00854, (ref-fp64): 0.00877 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2857146Z E0605 10:29:13.285000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00865, (ref-fp64): 0.00879 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2862058Z E0605 10:29:13.285000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00826, (ref-fp64): 0.00844 and shape=torch.Size([112, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2866425Z E0605 10:29:13.286000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00774, (ref-fp64): 0.00734 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2871008Z E0605 10:29:13.286000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00620, (ref-fp64): 0.00627 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2875421Z E0605 10:29:13.287000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00718, (ref-fp64): 0.00908 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2880156Z E0605 10:29:13.287000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.4.2.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00584, (ref-fp64): 0.00586 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2884740Z E0605 10:29:13.288000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01101, (ref-fp64): 0.01100 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2889373Z E0605 10:29:13.288000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00977, (ref-fp64): 0.00813 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2893381Z E0605 10:29:13.288000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01094, (ref-fp64): 0.01086 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2897895Z E0605 10:29:13.289000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00963, (ref-fp64): 0.00832 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2902053Z E0605 10:29:13.289000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01054, (ref-fp64): 0.01081 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2906346Z E0605 10:29:13.290000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00958, (ref-fp64): 0.01019 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2911142Z E0605 10:29:13.290000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00972, (ref-fp64): 0.00975 and shape=torch.Size([672, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2915834Z E0605 10:29:13.291000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.01012, (ref-fp64): 0.01019 and shape=torch.Size([672, 112, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2921233Z E0605 10:29:13.291000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00932, (ref-fp64): 0.00946 and shape=torch.Size([192, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2925662Z E0605 10:29:13.292000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00942, (ref-fp64): 0.00869 and shape=torch.Size([672]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2930434Z E0605 10:29:13.292000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00765, (ref-fp64): 0.00742 and shape=torch.Size([672, 28, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2934735Z E0605 10:29:13.293000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00779, (ref-fp64): 0.00830 and shape=torch.Size([28]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2939286Z E0605 10:29:13.293000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.0.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00642, (ref-fp64): 0.00646 and shape=torch.Size([28, 672, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2943844Z E0605 10:29:13.293000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01120, (ref-fp64): 0.01133 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2948234Z E0605 10:29:13.294000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00946, (ref-fp64): 0.00743 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2952601Z E0605 10:29:13.294000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01111, (ref-fp64): 0.01113 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2957461Z E0605 10:29:13.295000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00947, (ref-fp64): 0.00749 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2961490Z E0605 10:29:13.295000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01097, (ref-fp64): 0.01129 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2966578Z E0605 10:29:13.296000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00998, (ref-fp64): 0.00879 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2971254Z E0605 10:29:13.296000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00960, (ref-fp64): 0.00962 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2976259Z E0605 10:29:13.297000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00962, (ref-fp64): 0.00969 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2980656Z E0605 10:29:13.297000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00933, (ref-fp64): 0.00938 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2985107Z E0605 10:29:13.298000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00893, (ref-fp64): 0.00781 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.2989758Z E0605 10:29:13.298000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00651, (ref-fp64): 0.00644 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2993982Z E0605 10:29:13.298000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00815, (ref-fp64): 0.00859 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.2998729Z E0605 10:29:13.299000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.1.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00548, (ref-fp64): 0.00549 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3003115Z E0605 10:29:13.299000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01116, (ref-fp64): 0.01133 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3007919Z E0605 10:29:13.300000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00959, (ref-fp64): 0.00761 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3012306Z E0605 10:29:13.300000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01115, (ref-fp64): 0.01114 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3016540Z E0605 10:29:13.301000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00956, (ref-fp64): 0.00764 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3020754Z E0605 10:29:13.301000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01094, (ref-fp64): 0.01100 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3025044Z E0605 10:29:13.302000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01008, (ref-fp64): 0.00902 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3029674Z E0605 10:29:13.302000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00965, (ref-fp64): 0.00969 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3034335Z E0605 10:29:13.303000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00971, (ref-fp64): 0.00977 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3039282Z E0605 10:29:13.303000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00944 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3043528Z E0605 10:29:13.303000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00919, (ref-fp64): 0.00810 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3048933Z E0605 10:29:13.304000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00670, (ref-fp64): 0.00654 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3053026Z E0605 10:29:13.304000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00959, (ref-fp64): 0.00964 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3057709Z E0605 10:29:13.305000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.2.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00571, (ref-fp64): 0.00567 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3061957Z E0605 10:29:13.305000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01101, (ref-fp64): 0.01108 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3066349Z E0605 10:29:13.306000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00960, (ref-fp64): 0.00773 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3070699Z E0605 10:29:13.306000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01081, (ref-fp64): 0.01073 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3074989Z E0605 10:29:13.307000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00955, (ref-fp64): 0.00770 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3079233Z E0605 10:29:13.307000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01090, (ref-fp64): 0.01084 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3083717Z E0605 10:29:13.307000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01025, (ref-fp64): 0.00903 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3088560Z E0605 10:29:13.308000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00934, (ref-fp64): 0.00939 and shape=torch.Size([1152, 1, 5, 5]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3093160Z E0605 10:29:13.308000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.00968, (ref-fp64): 0.00972 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3098252Z E0605 10:29:13.309000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.00949, (ref-fp64): 0.00954 and shape=torch.Size([192, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3102362Z E0605 10:29:13.309000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00900, (ref-fp64): 0.00826 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3107291Z E0605 10:29:13.310000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00662, (ref-fp64): 0.00648 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3111435Z E0605 10:29:13.310000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.00949, (ref-fp64): 0.00944 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3116468Z E0605 10:29:13.311000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.5.3.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00613, (ref-fp64): 0.00609 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3120701Z E0605 10:29:13.311000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.01100, (ref-fp64): 0.01103 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3125493Z E0605 10:29:13.312000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00994, (ref-fp64): 0.00897 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3129853Z E0605 10:29:13.312000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn2.bias, passes_test: True, RMSE (res-fp64): 0.01104, (ref-fp64): 0.01105 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3134342Z E0605 10:29:13.313000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn2.weight, passes_test: True, RMSE (res-fp64): 0.00993, (ref-fp64): 0.00910 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3138749Z E0605 10:29:13.313000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.01085, (ref-fp64): 0.01088 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3142986Z E0605 10:29:13.313000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.01019, (ref-fp64): 0.01078 and shape=torch.Size([320]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3147570Z E0605 10:29:13.314000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.conv_dw.weight, passes_test: True, RMSE (res-fp64): 0.00999, (ref-fp64): 0.01013 and shape=torch.Size([1152, 1, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3152468Z E0605 10:29:13.314000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.conv_pw.weight, passes_test: True, RMSE (res-fp64): 0.01004, (ref-fp64): 0.01009 and shape=torch.Size([1152, 192, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3156866Z E0605 10:29:13.315000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.conv_pwl.weight, passes_test: True, RMSE (res-fp64): 0.01014, (ref-fp64): 0.01016 and shape=torch.Size([320, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3161233Z E0605 10:29:13.315000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.se.conv_expand.bias, passes_test: True, RMSE (res-fp64): 0.00984, (ref-fp64): 0.00937 and shape=torch.Size([1152]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3166190Z E0605 10:29:13.316000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.se.conv_expand.weight, passes_test: True, RMSE (res-fp64): 0.00871, (ref-fp64): 0.00849 and shape=torch.Size([1152, 48, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3170547Z E0605 10:29:13.316000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.se.conv_reduce.bias, passes_test: True, RMSE (res-fp64): 0.01010, (ref-fp64): 0.01003 and shape=torch.Size([48]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3175222Z E0605 10:29:13.317000 140570087031424 torch/_dynamo/utils.py:1482] key: blocks.6.0.se.conv_reduce.weight, passes_test: True, RMSE (res-fp64): 0.00806, (ref-fp64): 0.00806 and shape=torch.Size([48, 1152, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3179596Z E0605 10:29:13.317000 140570087031424 torch/_dynamo/utils.py:1482] key: bn1.bias, passes_test: True, RMSE (res-fp64): 0.01269, (ref-fp64): 0.01892 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3183949Z E0605 10:29:13.317000 140570087031424 torch/_dynamo/utils.py:1482] key: bn1.weight, passes_test: True, RMSE (res-fp64): 0.01075, (ref-fp64): 0.01528 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3188112Z E0605 10:29:13.318000 140570087031424 torch/_dynamo/utils.py:1482] key: bn2.bias, passes_test: True, RMSE (res-fp64): 0.01013, (ref-fp64): 0.01011 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3192647Z E0605 10:29:13.318000 140570087031424 torch/_dynamo/utils.py:1482] key: bn2.weight, passes_test: True, RMSE (res-fp64): 0.00995, (ref-fp64): 0.00957 and shape=torch.Size([1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3202633Z E0605 10:29:13.319000 140570087031424 torch/_dynamo/utils.py:1482] key: classifier.weight, passes_test: True, RMSE (res-fp64): 0.00336, (ref-fp64): 0.00361 and shape=torch.Size([1000, 1280]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:29:13.3207760Z E0605 10:29:13.320000 140570087031424 torch/_dynamo/utils.py:1482] key: conv_head.weight, passes_test: True, RMSE (res-fp64): 0.00996, (ref-fp64): 0.01000 and shape=torch.Size([1280, 320, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3212217Z E0605 10:29:13.320000 140570087031424 torch/_dynamo/utils.py:1482] key: conv_stem.weight, passes_test: True, RMSE (res-fp64): 0.01008, (ref-fp64): 0.01739 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:29:13.3523225Z pass
2024-06-05T10:29:13.3574179Z TIMING: entire_frame_compile:76.22887 code_gen:31.87204 inductor_compile:54.3521 backend_compile:61.72038
2024-06-05T10:29:13.3576232Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:21003 | FakeTensorMode.__torch_dispatch__:109611 | ProxyTorchDispatchMode.__torch_dispatch__:24017
2024-06-05T10:29:13.3577438Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique)
2024-06-05T10:29:21.0789258Z 
2024-06-05T10:29:23.7485038Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:29:23.7485984Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:29:23.7487254Z cuda train timm_regnet                        
2024-06-05T10:30:15.3408825Z pass
2024-06-05T10:30:15.3785874Z TIMING: entire_frame_compile:25.12717 code_gen:11.87413 inductor_compile:21.81311 backend_compile:18.71342
2024-06-05T10:30:15.3787284Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:9972 | FakeTensorMode.__torch_dispatch__:60472 | ProxyTorchDispatchMode.__torch_dispatch__:17103
2024-06-05T10:30:15.3788502Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique)
2024-06-05T10:30:20.5548310Z 
2024-06-05T10:30:22.4540818Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:30:22.4541347Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:30:22.4541823Z cuda train timm_resnest                       
2024-06-05T10:30:42.1689280Z W0605 10:30:42.168000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:42.2300241Z W0605 10:30:42.229000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:43.2891520Z W0605 10:30:43.288000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:43.3513749Z W0605 10:30:43.350000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:44.3927377Z W0605 10:30:44.392000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:44.4539140Z W0605 10:30:44.453000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:45.0398722Z W0605 10:30:45.039000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:45.1034805Z W0605 10:30:45.102000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:47.4708141Z W0605 10:30:47.470000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:48.1709694Z W0605 10:30:48.170000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:48.8692994Z W0605 10:30:48.868000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:49.3000052Z W0605 10:30:49.299000 139709661492992 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:30:51.2325835Z W0605 10:30:51.231000 139714734293632 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:31:15.3548590Z E0605 10:31:15.354000 139714734293632 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 4.85162, (ref-fp64): 5.08587 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3553421Z E0605 10:31:15.354000 139714734293632 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 4.84127, (ref-fp64): 5.07552 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3557955Z E0605 10:31:15.355000 139714734293632 torch/_dynamo/utils.py:1482] key: bn1.bias.grad, passes_test: True, RMSE (res-fp64): 0.38731, (ref-fp64): 0.31240 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3561979Z E0605 10:31:15.355000 139714734293632 torch/_dynamo/utils.py:1482] key: bn1.weight.grad, passes_test: True, RMSE (res-fp64): 0.43208, (ref-fp64): 0.34859 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3566646Z E0605 10:31:15.356000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.17536, (ref-fp64): 0.14320 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3570435Z E0605 10:31:15.356000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.1.bias.grad, passes_test: True, RMSE (res-fp64): 0.91083, (ref-fp64): 0.73906 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3574405Z E0605 10:31:15.357000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.37014, (ref-fp64): 0.29438 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3578618Z E0605 10:31:15.357000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.3.weight.grad, passes_test: True, RMSE (res-fp64): 0.16878, (ref-fp64): 0.12461 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3582659Z E0605 10:31:15.357000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.4.bias.grad, passes_test: True, RMSE (res-fp64): 0.93885, (ref-fp64): 0.68674 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3586773Z E0605 10:31:15.358000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.4.weight.grad, passes_test: True, RMSE (res-fp64): 0.91478, (ref-fp64): 0.69730 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3590964Z E0605 10:31:15.358000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.6.weight.grad, passes_test: True, RMSE (res-fp64): 0.21487, (ref-fp64): 0.17460 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3602446Z E0605 10:31:15.359000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00112, (ref-fp64): 0.00209 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3606811Z E0605 10:31:15.360000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00130, (ref-fp64): 0.00309 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3610937Z E0605 10:31:15.360000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.10787, (ref-fp64): 0.08913 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3615032Z E0605 10:31:15.361000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 0.02950, (ref-fp64): 0.02501 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3619279Z E0605 10:31:15.361000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00172 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3623376Z E0605 10:31:15.361000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00137, (ref-fp64): 0.00266 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3627473Z E0605 10:31:15.362000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00087, (ref-fp64): 0.00162 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3631877Z E0605 10:31:15.362000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00065, (ref-fp64): 0.00075 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3636014Z E0605 10:31:15.363000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00029, (ref-fp64): 0.00042 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3640352Z E0605 10:31:15.363000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00122, (ref-fp64): 0.00204 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3644674Z E0605 10:31:15.364000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00065, (ref-fp64): 0.00075 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3648880Z E0605 10:31:15.364000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00045, (ref-fp64): 0.00052 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3653073Z E0605 10:31:15.364000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc2.bias.grad, passes_test: True, RMSE (res-fp64): 0.00039, (ref-fp64): 0.00085 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3657234Z E0605 10:31:15.365000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00023, (ref-fp64): 0.00046 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3661257Z E0605 10:31:15.365000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00055, (ref-fp64): 0.00130 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3665512Z E0605 10:31:15.366000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.14525, (ref-fp64): 0.11906 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3669417Z E0605 10:31:15.366000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.2.bias.grad, passes_test: True, RMSE (res-fp64): 0.10787, (ref-fp64): 0.08913 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3673380Z E0605 10:31:15.366000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.10327, (ref-fp64): 0.08529 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3677344Z E0605 10:31:15.367000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias.grad, passes_test: True, RMSE (res-fp64): 0.00044, (ref-fp64): 0.00065 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3681465Z E0605 10:31:15.367000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00066 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3685705Z E0605 10:31:15.368000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.02630, (ref-fp64): 0.02491 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3690368Z E0605 10:31:15.368000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00957, (ref-fp64): 0.00836 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3694245Z E0605 10:31:15.368000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00027, (ref-fp64): 0.00036 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3698427Z E0605 10:31:15.369000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00078 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3702414Z E0605 10:31:15.369000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00021, (ref-fp64): 0.00042 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3708478Z E0605 10:31:15.370000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00011, (ref-fp64): 0.00021 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3713064Z E0605 10:31:15.370000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00031, (ref-fp64): 0.00049 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3723292Z E0605 10:31:15.371000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00011 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3728024Z E0605 10:31:15.372000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00014, (ref-fp64): 0.00032 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3734736Z E0605 10:31:15.372000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.02593, (ref-fp64): 0.02220 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3738862Z E0605 10:31:15.373000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.2.bias.grad, passes_test: True, RMSE (res-fp64): 0.02630, (ref-fp64): 0.02491 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3742783Z E0605 10:31:15.373000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.02828, (ref-fp64): 0.02617 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3750741Z E0605 10:31:15.374000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias.grad, passes_test: True, RMSE (res-fp64): 0.00396, (ref-fp64): 0.00378 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3755088Z E0605 10:31:15.375000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00225, (ref-fp64): 0.00218 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3770234Z E0605 10:31:15.376000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00009, (ref-fp64): 0.00018 and shape=torch.Size([512, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3780806Z E0605 10:31:15.377000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.fc2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00008 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3785203Z E0605 10:31:15.378000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00012 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3789910Z E0605 10:31:15.378000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00496, (ref-fp64): 0.00460 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3794058Z E0605 10:31:15.378000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.2.bias.grad, passes_test: True, RMSE (res-fp64): 0.00396, (ref-fp64): 0.00378 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3798720Z E0605 10:31:15.379000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.01052, (ref-fp64): 0.01052 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3845863Z E0605 10:31:15.384000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00108, (ref-fp64): 0.00107 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3852189Z E0605 10:31:15.384000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00588, (ref-fp64): 0.00598 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3856706Z E0605 10:31:15.385000 139714734293632 torch/_dynamo/utils.py:1482] key: bn1.bias, passes_test: True, RMSE (res-fp64): 0.00756, (ref-fp64): 0.00694 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3861106Z E0605 10:31:15.385000 139714734293632 torch/_dynamo/utils.py:1482] key: bn1.weight, passes_test: True, RMSE (res-fp64): 0.00756, (ref-fp64): 0.00743 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3865619Z E0605 10:31:15.386000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.0.weight, passes_test: True, RMSE (res-fp64): 0.00957, (ref-fp64): 0.00874 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3869871Z E0605 10:31:15.386000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.1.bias, passes_test: True, RMSE (res-fp64): 0.01120, (ref-fp64): 0.01174 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3874229Z E0605 10:31:15.386000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.1.weight, passes_test: True, RMSE (res-fp64): 0.00864, (ref-fp64): 0.00788 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3878824Z E0605 10:31:15.387000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.3.weight, passes_test: True, RMSE (res-fp64): 0.00740, (ref-fp64): 0.00632 and shape=torch.Size([32, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3883120Z E0605 10:31:15.387000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.4.bias, passes_test: True, RMSE (res-fp64): 0.00790, (ref-fp64): 0.00706 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3888011Z E0605 10:31:15.388000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.4.weight, passes_test: True, RMSE (res-fp64): 0.00790, (ref-fp64): 0.00706 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3892684Z E0605 10:31:15.388000 139714734293632 torch/_dynamo/utils.py:1482] key: conv1.6.weight, passes_test: True, RMSE (res-fp64): 0.00801, (ref-fp64): 0.00734 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3905791Z E0605 10:31:15.390000 139714734293632 torch/_dynamo/utils.py:1482] key: fc.weight, passes_test: True, RMSE (res-fp64): 0.00112, (ref-fp64): 0.00105 and shape=torch.Size([1000, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3910089Z E0605 10:31:15.390000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00473, (ref-fp64): 0.00558 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3914338Z E0605 10:31:15.391000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00447, (ref-fp64): 0.00545 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3918603Z E0605 10:31:15.391000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.00592, (ref-fp64): 0.00588 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3923162Z E0605 10:31:15.391000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00647, (ref-fp64): 0.01226 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3927849Z E0605 10:31:15.392000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00441, (ref-fp64): 0.00500 and shape=torch.Size([64, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3932082Z E0605 10:31:15.392000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn0.bias, passes_test: True, RMSE (res-fp64): 0.00469, (ref-fp64): 0.00730 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3936287Z E0605 10:31:15.393000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn0.weight, passes_test: True, RMSE (res-fp64): 0.00450, (ref-fp64): 0.00690 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3940574Z E0605 10:31:15.393000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00436, (ref-fp64): 0.00540 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3944959Z E0605 10:31:15.394000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00436, (ref-fp64): 0.00537 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3949340Z E0605 10:31:15.394000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.conv.weight, passes_test: True, RMSE (res-fp64): 0.00384, (ref-fp64): 0.00503 and shape=torch.Size([128, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.3953548Z E0605 10:31:15.394000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00436, (ref-fp64): 0.00540 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3958168Z E0605 10:31:15.395000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00434, (ref-fp64): 0.00514 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3962542Z E0605 10:31:15.395000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00532, (ref-fp64): 0.00764 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3967384Z E0605 10:31:15.396000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00343, (ref-fp64): 0.00421 and shape=torch.Size([128, 32, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3972206Z E0605 10:31:15.396000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00287, (ref-fp64): 0.00560 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3976375Z E0605 10:31:15.397000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.00622, (ref-fp64): 0.00597 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3980734Z E0605 10:31:15.397000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.2.bias, passes_test: True, RMSE (res-fp64): 0.00592, (ref-fp64): 0.00588 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3985228Z E0605 10:31:15.398000 139714734293632 torch/_dynamo/utils.py:1482] key: layer1.0.downsample.2.weight, passes_test: True, RMSE (res-fp64): 0.00559, (ref-fp64): 0.01006 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3989522Z E0605 10:31:15.398000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00356, (ref-fp64): 0.00493 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3993746Z E0605 10:31:15.398000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00337, (ref-fp64): 0.00506 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.3998205Z E0605 10:31:15.399000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.00270, (ref-fp64): 0.00260 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4003165Z E0605 10:31:15.399000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00347, (ref-fp64): 0.01061 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4007942Z E0605 10:31:15.400000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00314, (ref-fp64): 0.00380 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4012218Z E0605 10:31:15.400000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn0.bias, passes_test: True, RMSE (res-fp64): 0.00345, (ref-fp64): 0.00585 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4017002Z E0605 10:31:15.401000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn0.weight, passes_test: True, RMSE (res-fp64): 0.00301, (ref-fp64): 0.00578 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4021295Z E0605 10:31:15.401000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00436, (ref-fp64): 0.00416 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4025289Z E0605 10:31:15.402000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00429, (ref-fp64): 0.00414 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4030402Z E0605 10:31:15.402000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.conv.weight, passes_test: True, RMSE (res-fp64): 0.00317, (ref-fp64): 0.00444 and shape=torch.Size([256, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4034273Z E0605 10:31:15.403000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00436, (ref-fp64): 0.00416 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4039094Z E0605 10:31:15.403000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00421, (ref-fp64): 0.00401 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4043561Z E0605 10:31:15.403000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00587, (ref-fp64): 0.00757 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4048442Z E0605 10:31:15.404000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00483, (ref-fp64): 0.00594 and shape=torch.Size([256, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4053136Z E0605 10:31:15.404000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00169, (ref-fp64): 0.00403 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4057727Z E0605 10:31:15.405000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.00308, (ref-fp64): 0.00290 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4061972Z E0605 10:31:15.405000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.2.bias, passes_test: True, RMSE (res-fp64): 0.00270, (ref-fp64): 0.00260 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4066251Z E0605 10:31:15.406000 139714734293632 torch/_dynamo/utils.py:1482] key: layer2.0.downsample.2.weight, passes_test: True, RMSE (res-fp64): 0.00252, (ref-fp64): 0.00781 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4070514Z E0605 10:31:15.406000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00258, (ref-fp64): 0.00446 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4074825Z E0605 10:31:15.407000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00244, (ref-fp64): 0.00427 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4079082Z E0605 10:31:15.407000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.00221, (ref-fp64): 0.00216 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4083494Z E0605 10:31:15.407000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00253, (ref-fp64): 0.00763 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4088450Z E0605 10:31:15.408000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00198, (ref-fp64): 0.00303 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4092615Z E0605 10:31:15.408000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.bn0.bias, passes_test: True, RMSE (res-fp64): 0.00415, (ref-fp64): 0.00526 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4096881Z E0605 10:31:15.409000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.bn0.weight, passes_test: True, RMSE (res-fp64): 0.00386, (ref-fp64): 0.00527 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4101228Z E0605 10:31:15.409000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00328, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4105591Z E0605 10:31:15.410000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00327, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4110469Z E0605 10:31:15.410000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.conv.weight, passes_test: True, RMSE (res-fp64): 0.00284, (ref-fp64): 0.00372 and shape=torch.Size([512, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4114808Z E0605 10:31:15.411000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00328, (ref-fp64): 0.00366 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4119547Z E0605 10:31:15.411000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00310, (ref-fp64): 0.00356 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4124012Z E0605 10:31:15.411000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00309, (ref-fp64): 0.00562 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4129449Z E0605 10:31:15.412000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00250, (ref-fp64): 0.00404 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4133751Z E0605 10:31:15.412000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00292 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4138523Z E0605 10:31:15.413000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.00223, (ref-fp64): 0.00217 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4142746Z E0605 10:31:15.413000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.2.bias, passes_test: True, RMSE (res-fp64): 0.00221, (ref-fp64): 0.00216 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4154856Z E0605 10:31:15.414000 139714734293632 torch/_dynamo/utils.py:1482] key: layer3.0.downsample.2.weight, passes_test: True, RMSE (res-fp64): 0.00188, (ref-fp64): 0.00596 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4157787Z E0605 10:31:15.414000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00324, (ref-fp64): 0.00455 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4160373Z E0605 10:31:15.415000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00317, (ref-fp64): 0.00426 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4163366Z E0605 10:31:15.415000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.bias, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00164 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4166055Z E0605 10:31:15.415000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.bn3.weight, passes_test: True, RMSE (res-fp64): 0.00186, (ref-fp64): 0.00646 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4169285Z E0605 10:31:15.416000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv1.weight, passes_test: True, RMSE (res-fp64): 0.00212, (ref-fp64): 0.00291 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4173414Z E0605 10:31:15.416000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.bn0.bias, passes_test: True, RMSE (res-fp64): 0.00299, (ref-fp64): 0.00537 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4177399Z E0605 10:31:15.417000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.bn0.weight, passes_test: True, RMSE (res-fp64): 0.00283, (ref-fp64): 0.00540 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4181899Z E0605 10:31:15.417000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.bn1.bias, passes_test: True, RMSE (res-fp64): 0.00333, (ref-fp64): 0.00474 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4185991Z E0605 10:31:15.418000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.bn1.weight, passes_test: True, RMSE (res-fp64): 0.00332, (ref-fp64): 0.00471 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4198565Z E0605 10:31:15.419000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.conv.weight, passes_test: True, RMSE (res-fp64): 0.00264, (ref-fp64): 0.00371 and shape=torch.Size([1024, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4202888Z E0605 10:31:15.419000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.fc1.bias, passes_test: True, RMSE (res-fp64): 0.00333, (ref-fp64): 0.00474 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4207956Z E0605 10:31:15.420000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.fc1.weight, passes_test: True, RMSE (res-fp64): 0.00314, (ref-fp64): 0.00448 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4212520Z E0605 10:31:15.420000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.fc2.bias, passes_test: True, RMSE (res-fp64): 0.00255, (ref-fp64): 0.00538 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4216775Z E0605 10:31:15.421000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv2.fc2.weight, passes_test: True, RMSE (res-fp64): 0.00221, (ref-fp64): 0.00417 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4223296Z E0605 10:31:15.421000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.conv3.weight, passes_test: True, RMSE (res-fp64): 0.00064, (ref-fp64): 0.00237 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4234785Z E0605 10:31:15.423000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.1.weight, passes_test: True, RMSE (res-fp64): 0.00171, (ref-fp64): 0.00167 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:31:15.4239033Z E0605 10:31:15.423000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.2.bias, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00164 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4243428Z E0605 10:31:15.423000 139714734293632 torch/_dynamo/utils.py:1482] key: layer4.0.downsample.2.weight, passes_test: True, RMSE (res-fp64): 0.00103, (ref-fp64): 0.00481 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:15.4387883Z pass
2024-06-05T10:31:15.4454891Z TIMING: entire_frame_compile:33.64767 code_gen:22.32686 inductor_compile:31.51859 backend_compile:28.83127
2024-06-05T10:31:15.4456384Z STATS: call_* op count: 419 | FakeTensor.__torch_dispatch__:8090 | FakeTensorMode.__torch_dispatch__:44789 | ProxyTorchDispatchMode.__torch_dispatch__:9820
2024-06-05T10:31:15.4457566Z Dynamo produced 3 graphs covering 419 ops with 7 graph breaks (5 unique)
2024-06-05T10:31:20.7227581Z 
2024-06-05T10:31:22.2746616Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:31:22.2747762Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:31:22.2749063Z cuda train timm_vision_transformer            
2024-06-05T10:31:50.2348676Z E0605 10:31:50.233000 140245069632128 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00049, (ref-fp64): 0.00049 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:31:50.2960058Z pass
2024-06-05T10:31:50.3137822Z TIMING: entire_frame_compile:16.67235 code_gen:8.24395 inductor_compile:13.9747 backend_compile:14.93778
2024-06-05T10:31:50.3139233Z STATS: call_* op count: 355 | FakeTensor.__torch_dispatch__:6172 | FakeTensorMode.__torch_dispatch__:35258 | ProxyTorchDispatchMode.__torch_dispatch__:10943
2024-06-05T10:31:50.3140510Z Dynamo produced 2 graphs covering 355 ops with 6 graph breaks (5 unique)
2024-06-05T10:31:54.8811938Z 
2024-06-05T10:32:07.1126182Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:32:07.1127050Z loading model: 0it [00:12, ?it/s]
2024-06-05T10:32:07.1129693Z cuda train timm_vision_transformer_large      
2024-06-05T10:32:07.1130705Z pass_due_to_skip
2024-06-05T10:32:07.3753458Z TIMING:
2024-06-05T10:32:07.3753851Z STATS: call_* op count: 0
2024-06-05T10:32:07.3754484Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-06-05T10:32:10.7594873Z 
2024-06-05T10:32:12.7335261Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:32:12.7335783Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:32:12.7336286Z cuda train timm_vovnet                        
2024-06-05T10:32:48.8258030Z E0605 10:32:48.824000 140075065684608 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.08044, (ref-fp64): 0.08034 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8270528Z E0605 10:32:48.826000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_concat.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00430, (ref-fp64): 0.00442 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8275459Z E0605 10:32:48.827000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_concat.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00266, (ref-fp64): 0.00264 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8280902Z E0605 10:32:48.827000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_concat.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00073, (ref-fp64): 0.00074 and shape=torch.Size([256, 768, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8285789Z E0605 10:32:48.828000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.0.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.01118, (ref-fp64): 0.01151 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8290638Z E0605 10:32:48.828000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.0.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00267, (ref-fp64): 0.00276 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8295434Z E0605 10:32:48.829000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00179, (ref-fp64): 0.00182 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8300005Z E0605 10:32:48.829000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.1.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00986, (ref-fp64): 0.01070 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8304368Z E0605 10:32:48.829000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.1.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00257, (ref-fp64): 0.00280 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8309302Z E0605 10:32:48.830000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00159, (ref-fp64): 0.00169 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8313608Z E0605 10:32:48.830000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.2.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00889, (ref-fp64): 0.00956 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8318025Z E0605 10:32:48.831000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.2.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00211, (ref-fp64): 0.00224 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8322934Z E0605 10:32:48.831000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00159 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8327770Z E0605 10:32:48.832000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.3.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00585, (ref-fp64): 0.00639 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8332291Z E0605 10:32:48.832000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.3.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00159, (ref-fp64): 0.00163 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8337044Z E0605 10:32:48.833000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00106 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8341339Z E0605 10:32:48.833000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.4.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00468, (ref-fp64): 0.00482 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8348256Z E0605 10:32:48.834000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.0.blocks.0.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00084, (ref-fp64): 0.00086 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8352622Z E0605 10:32:48.834000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_concat.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00115, (ref-fp64): 0.00115 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8357138Z E0605 10:32:48.835000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_concat.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00178, (ref-fp64): 0.00183 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8362215Z E0605 10:32:48.835000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_concat.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00090, (ref-fp64): 0.00091 and shape=torch.Size([512, 1056, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8367156Z E0605 10:32:48.836000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.0.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00333, (ref-fp64): 0.00349 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8371787Z E0605 10:32:48.836000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.0.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00285, (ref-fp64): 0.00298 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8376975Z E0605 10:32:48.837000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00165, (ref-fp64): 0.00172 and shape=torch.Size([160, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8381516Z E0605 10:32:48.837000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.1.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00288, (ref-fp64): 0.00290 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8385915Z E0605 10:32:48.838000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.1.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00238, (ref-fp64): 0.00236 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8390898Z E0605 10:32:48.838000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00186, (ref-fp64): 0.00187 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8395228Z E0605 10:32:48.839000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.2.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00218, (ref-fp64): 0.00213 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8399832Z E0605 10:32:48.839000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.2.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00241, (ref-fp64): 0.00238 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8404887Z E0605 10:32:48.840000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00162, (ref-fp64): 0.00158 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8409862Z E0605 10:32:48.840000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.3.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.00174, (ref-fp64): 0.00181 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8414342Z E0605 10:32:48.840000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.3.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00241, (ref-fp64): 0.00249 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8418924Z E0605 10:32:48.841000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00142, (ref-fp64): 0.00148 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8425389Z E0605 10:32:48.842000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.4.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00189, (ref-fp64): 0.00190 and shape=torch.Size([160]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8430125Z E0605 10:32:48.842000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.1.blocks.0.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00116, (ref-fp64): 0.00118 and shape=torch.Size([160, 160, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8436541Z E0605 10:32:48.843000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_concat.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00131, (ref-fp64): 0.00130 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8443764Z E0605 10:32:48.843000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_concat.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00069, (ref-fp64): 0.00069 and shape=torch.Size([768, 1472, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8450660Z E0605 10:32:48.844000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.0.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00270, (ref-fp64): 0.00270 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8456568Z E0605 10:32:48.845000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00120, (ref-fp64): 0.00118 and shape=torch.Size([192, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8463116Z E0605 10:32:48.845000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.1.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00218, (ref-fp64): 0.00214 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8468251Z E0605 10:32:48.846000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00160, (ref-fp64): 0.00154 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8474658Z E0605 10:32:48.847000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.2.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00210, (ref-fp64): 0.00217 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8479563Z E0605 10:32:48.847000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00148, (ref-fp64): 0.00154 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8489000Z E0605 10:32:48.848000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00111, (ref-fp64): 0.00110 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8497585Z E0605 10:32:48.849000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.0.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00083, (ref-fp64): 0.00082 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8510072Z E0605 10:32:48.850000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_concat.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00042, (ref-fp64): 0.00042 and shape=torch.Size([768, 1728, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8516527Z E0605 10:32:48.851000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.0.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00257, (ref-fp64): 0.00261 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8524623Z E0605 10:32:48.851000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00089, (ref-fp64): 0.00089 and shape=torch.Size([192, 768, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8531361Z E0605 10:32:48.852000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.1.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00221, (ref-fp64): 0.00216 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8536417Z E0605 10:32:48.853000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00152, (ref-fp64): 0.00150 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8542786Z E0605 10:32:48.853000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.2.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00180, (ref-fp64): 0.00173 and shape=torch.Size([192]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8547570Z E0605 10:32:48.854000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00122, (ref-fp64): 0.00122 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8556236Z E0605 10:32:48.855000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00085, (ref-fp64): 0.00088 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8565086Z E0605 10:32:48.856000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.2.blocks.1.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00058, (ref-fp64): 0.00059 and shape=torch.Size([192, 192, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8580149Z E0605 10:32:48.857000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_concat.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00022, (ref-fp64): 0.00022 and shape=torch.Size([1024, 1888, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8593419Z E0605 10:32:48.858000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00050, (ref-fp64): 0.00049 and shape=torch.Size([224, 768, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8602242Z E0605 10:32:48.859000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00078 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8611258Z E0605 10:32:48.860000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00060, (ref-fp64): 0.00060 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8620268Z E0605 10:32:48.861000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00042, (ref-fp64): 0.00043 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8628976Z E0605 10:32:48.862000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.0.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00025, (ref-fp64): 0.00026 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8654456Z E0605 10:32:48.864000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.1.conv_mid.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00023 and shape=torch.Size([224, 1024, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8663030Z E0605 10:32:48.865000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.1.conv_mid.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00040, (ref-fp64): 0.00040 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8671693Z E0605 10:32:48.866000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.1.conv_mid.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00024 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8680219Z E0605 10:32:48.867000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.1.conv_mid.3.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00020, (ref-fp64): 0.00020 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8689421Z E0605 10:32:48.868000 140075065684608 torch/_dynamo/utils.py:1482] key: stages.3.blocks.1.conv_mid.4.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00013, (ref-fp64): 0.00012 and shape=torch.Size([224, 224, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8693572Z E0605 10:32:48.868000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.0.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.01238, (ref-fp64): 0.01339 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8698138Z E0605 10:32:48.869000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.0.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00380, (ref-fp64): 0.00410 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8702779Z E0605 10:32:48.869000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.0.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.01151, (ref-fp64): 0.01161 and shape=torch.Size([64, 3, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8707300Z E0605 10:32:48.870000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.1.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.01293, (ref-fp64): 0.01317 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8711884Z E0605 10:32:48.870000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.1.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00358, (ref-fp64): 0.00364 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8716598Z E0605 10:32:48.871000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.1.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00256, (ref-fp64): 0.00258 and shape=torch.Size([64, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.8721037Z E0605 10:32:48.871000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.2.bn.bias.grad, passes_test: True, RMSE (res-fp64): 0.01178, (ref-fp64): 0.01225 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8725755Z E0605 10:32:48.872000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.2.bn.weight.grad, passes_test: True, RMSE (res-fp64): 0.00264, (ref-fp64): 0.00282 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:32:48.8730853Z E0605 10:32:48.872000 140075065684608 torch/_dynamo/utils.py:1482] key: stem.2.conv.weight.grad, passes_test: True, RMSE (res-fp64): 0.00255, (ref-fp64): 0.00259 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:32:48.9227709Z pass
2024-06-05T10:32:48.9406349Z TIMING: entire_frame_compile:15.15728 code_gen:13.63022 inductor_compile:18.95073 backend_compile:11.73618
2024-06-05T10:32:48.9409148Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:4772 | FakeTensorMode.__torch_dispatch__:30297 | ProxyTorchDispatchMode.__torch_dispatch__:8526
2024-06-05T10:32:48.9411545Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique)
2024-06-05T10:32:53.4942699Z 
2024-06-05T10:32:55.3880749Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:32:55.3881468Z loading model: 0it [00:01, ?it/s]
2024-06-05T10:32:55.3882222Z cuda train torch_multimodal_clip              
2024-06-05T10:33:31.5597631Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_302)
2024-06-05T10:33:37.2233005Z W0605 10:33:37.222000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2234607Z W0605 10:33:37.222000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2310487Z W0605 10:33:37.230000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2312063Z W0605 10:33:37.230000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2313597Z W0605 10:33:37.230000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2992907Z W0605 10:33:37.298000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.2994488Z W0605 10:33:37.298000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.3139181Z W0605 10:33:37.313000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.3140803Z W0605 10:33:37.313000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:37.3142316Z W0605 10:33:37.313000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] d2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:41.8592695Z W0605 10:33:41.858000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:41.8594252Z W0605 10:33:41.858000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:41.8733076Z W0605 10:33:41.872000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:41.8734799Z W0605 10:33:41.872000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:41.8736338Z W0605 10:33:41.873000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] q0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:42.0190900Z W0605 10:33:42.018000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:42.0192658Z W0605 10:33:42.018000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:42.0194178Z W0605 10:33:42.018000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] z1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:46.3916640Z W0605 10:33:46.391000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x0 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:46.3918488Z W0605 10:33:46.391000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] r2 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:46.3920036Z W0605 10:33:46.391000 140236533487360 torch/fx/experimental/symbolic_shapes.py:4424] [3/0_1] x1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:33:49.9494001Z W0605 10:33:49.948000 140242226967168 torch/_logging/_internal.py:1033] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:35:02.2436645Z E0605 10:35:02.242000 140242226967168 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00616, (ref-fp64): 0.00614 and shape=torch.Size([4, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.2439881Z E0605 10:35:02.243000 140242226967168 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.04645, (ref-fp64): 0.04851 and shape=torch.Size([32, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.2914150Z E0605 10:35:02.290000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.self_attn.out_proj.weight.grad, passes_test: True, RMSE (res-fp64): 0.00047, (ref-fp64): 0.00054 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3197032Z E0605 10:35:02.319000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.positional_embedding.grad, passes_test: True, RMSE (res-fp64): 0.00024, (ref-fp64): 0.00026 and shape=torch.Size([77, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3310274Z E0605 10:35:02.330000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.token_embedding.weight.grad, passes_test: True, RMSE (res-fp64): 0.00001, (ref-fp64): 0.00001 and shape=torch.Size([49408, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3316077Z E0605 10:35:02.331000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.cls_token_embedding, passes_test: True, RMSE (res-fp64): 0.00338, (ref-fp64): 0.00330 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3329493Z E0605 10:35:02.332000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.conv.weight, passes_test: True, RMSE (res-fp64): 0.01129, (ref-fp64): 0.01095 and shape=torch.Size([768, 3, 32, 32]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3335179Z E0605 10:35:02.333000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00965, (ref-fp64): 0.00970 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3348048Z E0605 10:35:02.334000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01044, (ref-fp64): 0.01082 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3353314Z E0605 10:35:02.334000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00892, (ref-fp64): 0.00896 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3366674Z E0605 10:35:02.336000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00914, (ref-fp64): 0.00911 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3371162Z E0605 10:35:02.336000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01015, (ref-fp64): 0.00998 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3376061Z E0605 10:35:02.337000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00954, (ref-fp64): 0.01019 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3380478Z E0605 10:35:02.337000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00865, (ref-fp64): 0.00869 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3385201Z E0605 10:35:02.338000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00966, (ref-fp64): 0.01018 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3389542Z E0605 10:35:02.338000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01467, (ref-fp64): 0.01450 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3399902Z E0605 10:35:02.339000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01307, (ref-fp64): 0.01287 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3404568Z E0605 10:35:02.339000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00937, (ref-fp64): 0.00899 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3409697Z E0605 10:35:02.340000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.0.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00893, (ref-fp64): 0.00878 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3414716Z E0605 10:35:02.341000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00926, (ref-fp64): 0.00971 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3427166Z E0605 10:35:02.342000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01049, (ref-fp64): 0.01068 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3432030Z E0605 10:35:02.342000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00952, (ref-fp64): 0.00934 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3444684Z E0605 10:35:02.344000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00935, (ref-fp64): 0.00932 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3449675Z E0605 10:35:02.344000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01053, (ref-fp64): 0.01025 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3454999Z E0605 10:35:02.345000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01076, (ref-fp64): 0.01030 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3459768Z E0605 10:35:02.345000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00999, (ref-fp64): 0.00980 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3464701Z E0605 10:35:02.346000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01080, (ref-fp64): 0.01146 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3469664Z E0605 10:35:02.346000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01543, (ref-fp64): 0.01549 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3479869Z E0605 10:35:02.347000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01410, (ref-fp64): 0.01408 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3484831Z E0605 10:35:02.348000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00968, (ref-fp64): 0.00933 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3490562Z E0605 10:35:02.348000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.1.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00897, (ref-fp64): 0.00870 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3495494Z E0605 10:35:02.349000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00642, (ref-fp64): 0.00700 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3508089Z E0605 10:35:02.350000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00715, (ref-fp64): 0.00756 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3513021Z E0605 10:35:02.350000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00460, (ref-fp64): 0.00461 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3525858Z E0605 10:35:02.352000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00555, (ref-fp64): 0.00558 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3531020Z E0605 10:35:02.352000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01110, (ref-fp64): 0.01120 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3535921Z E0605 10:35:02.353000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01205, (ref-fp64): 0.01242 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3540665Z E0605 10:35:02.353000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00416, (ref-fp64): 0.00478 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3545659Z E0605 10:35:02.354000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00579, (ref-fp64): 0.00626 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3550666Z E0605 10:35:02.354000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00869, (ref-fp64): 0.00865 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3560596Z E0605 10:35:02.355000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01148, (ref-fp64): 0.01148 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3566857Z E0605 10:35:02.356000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00452, (ref-fp64): 0.00460 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3572118Z E0605 10:35:02.356000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.10.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00476, (ref-fp64): 0.00483 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3577363Z E0605 10:35:02.357000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00443, (ref-fp64): 0.00498 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3589982Z E0605 10:35:02.358000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00473, (ref-fp64): 0.00510 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3594930Z E0605 10:35:02.359000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00058, (ref-fp64): 0.00058 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3607735Z E0605 10:35:02.360000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00182, (ref-fp64): 0.00182 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3612887Z E0605 10:35:02.360000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01060, (ref-fp64): 0.01046 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3617868Z E0605 10:35:02.361000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01211, (ref-fp64): 0.01244 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3622464Z E0605 10:35:02.361000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00345, (ref-fp64): 0.00377 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3627422Z E0605 10:35:02.362000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00379, (ref-fp64): 0.00395 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3632280Z E0605 10:35:02.362000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00876, (ref-fp64): 0.00869 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3642707Z E0605 10:35:02.363000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01142, (ref-fp64): 0.01138 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3647977Z E0605 10:35:02.364000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00075, (ref-fp64): 0.00076 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3653357Z E0605 10:35:02.364000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.11.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00268, (ref-fp64): 0.00267 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3658349Z E0605 10:35:02.365000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00977, (ref-fp64): 0.00978 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3670975Z E0605 10:35:02.366000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01071, (ref-fp64): 0.01081 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3675852Z E0605 10:35:02.367000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00935, (ref-fp64): 0.00932 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3688551Z E0605 10:35:02.368000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00911, (ref-fp64): 0.00910 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3693501Z E0605 10:35:02.368000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01050, (ref-fp64): 0.01087 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3698316Z E0605 10:35:02.369000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01136, (ref-fp64): 0.01109 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3703163Z E0605 10:35:02.369000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00945, (ref-fp64): 0.00929 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3708028Z E0605 10:35:02.370000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01022, (ref-fp64): 0.01008 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3712985Z E0605 10:35:02.370000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01507, (ref-fp64): 0.01503 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3723229Z E0605 10:35:02.371000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01412, (ref-fp64): 0.01409 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3728571Z E0605 10:35:02.372000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00923, (ref-fp64): 0.00922 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3734120Z E0605 10:35:02.372000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.2.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00862, (ref-fp64): 0.00860 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3739108Z E0605 10:35:02.373000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01023, (ref-fp64): 0.01039 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3752123Z E0605 10:35:02.374000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01086, (ref-fp64): 0.01098 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3757196Z E0605 10:35:02.375000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00979, (ref-fp64): 0.00967 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3770902Z E0605 10:35:02.376000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00877, (ref-fp64): 0.00880 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3775121Z E0605 10:35:02.377000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01119, (ref-fp64): 0.01105 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3779963Z E0605 10:35:02.377000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01186, (ref-fp64): 0.01176 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3784976Z E0605 10:35:02.378000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.norm2.bias, passes_test: True, RMSE (res-fp64): 0.01007, (ref-fp64): 0.00969 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3790101Z E0605 10:35:02.378000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01055, (ref-fp64): 0.01049 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3795025Z E0605 10:35:02.379000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01435, (ref-fp64): 0.01435 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3805281Z E0605 10:35:02.380000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01389, (ref-fp64): 0.01388 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3810087Z E0605 10:35:02.380000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01019 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3816130Z E0605 10:35:02.381000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.3.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00913, (ref-fp64): 0.00914 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3820821Z E0605 10:35:02.381000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01050, (ref-fp64): 0.01091 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3833175Z E0605 10:35:02.382000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01100, (ref-fp64): 0.01127 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3838802Z E0605 10:35:02.383000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00959, (ref-fp64): 0.00965 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3851792Z E0605 10:35:02.384000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00850, (ref-fp64): 0.00853 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3856875Z E0605 10:35:02.385000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01125, (ref-fp64): 0.01129 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3861887Z E0605 10:35:02.385000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01149, (ref-fp64): 0.01207 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3867269Z E0605 10:35:02.386000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.norm2.bias, passes_test: True, RMSE (res-fp64): 0.01076, (ref-fp64): 0.01057 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3872576Z E0605 10:35:02.386000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01141, (ref-fp64): 0.01101 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3878145Z E0605 10:35:02.387000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01285, (ref-fp64): 0.01291 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3889727Z E0605 10:35:02.388000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01341, (ref-fp64): 0.01339 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3893214Z E0605 10:35:02.388000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00910, (ref-fp64): 0.00954 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3898755Z E0605 10:35:02.389000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.4.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00833, (ref-fp64): 0.00863 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3903733Z E0605 10:35:02.389000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01239, (ref-fp64): 0.01264 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3916645Z E0605 10:35:02.391000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01182, (ref-fp64): 0.01214 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3921654Z E0605 10:35:02.391000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00926, (ref-fp64): 0.00903 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3934521Z E0605 10:35:02.393000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00821, (ref-fp64): 0.00819 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3939565Z E0605 10:35:02.393000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01060, (ref-fp64): 0.01081 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3944547Z E0605 10:35:02.394000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01175, (ref-fp64): 0.01223 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3949700Z E0605 10:35:02.394000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.norm2.bias, passes_test: True, RMSE (res-fp64): 0.01347, (ref-fp64): 0.01310 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3954615Z E0605 10:35:02.395000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01296, (ref-fp64): 0.01301 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3959792Z E0605 10:35:02.395000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01121, (ref-fp64): 0.01132 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3970373Z E0605 10:35:02.396000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01288, (ref-fp64): 0.01293 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3975482Z E0605 10:35:02.397000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00965, (ref-fp64): 0.00951 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.3981031Z E0605 10:35:02.397000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.5.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00876, (ref-fp64): 0.00867 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3986217Z E0605 10:35:02.398000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01286, (ref-fp64): 0.01313 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.3999014Z E0605 10:35:02.399000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01206, (ref-fp64): 0.01239 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4004304Z E0605 10:35:02.399000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00957 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4016982Z E0605 10:35:02.401000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00844, (ref-fp64): 0.00850 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4021841Z E0605 10:35:02.401000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01113, (ref-fp64): 0.01176 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4026986Z E0605 10:35:02.402000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01201, (ref-fp64): 0.01257 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4032006Z E0605 10:35:02.402000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.norm2.bias, passes_test: True, RMSE (res-fp64): 0.01299, (ref-fp64): 0.01287 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4037158Z E0605 10:35:02.403000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01300, (ref-fp64): 0.01322 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4042476Z E0605 10:35:02.403000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.01071, (ref-fp64): 0.01078 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4052719Z E0605 10:35:02.404000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01260, (ref-fp64): 0.01263 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4057482Z E0605 10:35:02.405000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00941, (ref-fp64): 0.00934 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4063181Z E0605 10:35:02.405000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.6.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00857, (ref-fp64): 0.00857 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4067994Z E0605 10:35:02.406000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01292, (ref-fp64): 0.01324 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4080894Z E0605 10:35:02.407000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01210, (ref-fp64): 0.01238 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4086661Z E0605 10:35:02.408000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00841, (ref-fp64): 0.00832 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4099265Z E0605 10:35:02.409000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00782, (ref-fp64): 0.00780 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4104394Z E0605 10:35:02.409000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01234, (ref-fp64): 0.01172 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4109944Z E0605 10:35:02.410000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01322, (ref-fp64): 0.01305 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4114874Z E0605 10:35:02.411000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.norm2.bias, passes_test: True, RMSE (res-fp64): 0.01136, (ref-fp64): 0.01113 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4120248Z E0605 10:35:02.411000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.norm2.weight, passes_test: True, RMSE (res-fp64): 0.01183, (ref-fp64): 0.01196 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4125544Z E0605 10:35:02.412000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00977, (ref-fp64): 0.00981 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4135832Z E0605 10:35:02.413000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01239, (ref-fp64): 0.01239 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4140844Z E0605 10:35:02.413000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00837, (ref-fp64): 0.00822 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4146269Z E0605 10:35:02.414000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.7.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00769, (ref-fp64): 0.00764 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4151364Z E0605 10:35:02.414000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.linear1.bias, passes_test: True, RMSE (res-fp64): 0.01111, (ref-fp64): 0.01171 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4164366Z E0605 10:35:02.415000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.linear1.weight, passes_test: True, RMSE (res-fp64): 0.01083, (ref-fp64): 0.01126 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4169654Z E0605 10:35:02.416000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00731, (ref-fp64): 0.00752 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4182104Z E0605 10:35:02.417000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00714, (ref-fp64): 0.00723 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4187134Z E0605 10:35:02.418000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01153, (ref-fp64): 0.01126 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4192175Z E0605 10:35:02.418000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01321, (ref-fp64): 0.01309 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4197225Z E0605 10:35:02.419000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00869, (ref-fp64): 0.00883 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4202440Z E0605 10:35:02.419000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00992, (ref-fp64): 0.01026 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4208096Z E0605 10:35:02.420000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00892, (ref-fp64): 0.00888 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4218464Z E0605 10:35:02.421000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01193, (ref-fp64): 0.01196 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4223643Z E0605 10:35:02.421000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00746, (ref-fp64): 0.00738 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4229294Z E0605 10:35:02.422000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.8.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00711, (ref-fp64): 0.00704 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4234573Z E0605 10:35:02.423000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00891, (ref-fp64): 0.00946 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4247523Z E0605 10:35:02.424000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00922, (ref-fp64): 0.00965 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4252902Z E0605 10:35:02.424000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00647, (ref-fp64): 0.00654 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4265377Z E0605 10:35:02.426000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00640, (ref-fp64): 0.00643 and shape=torch.Size([768, 3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4270404Z E0605 10:35:02.426000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.norm1.bias, passes_test: True, RMSE (res-fp64): 0.01201, (ref-fp64): 0.01146 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4275751Z E0605 10:35:02.427000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.norm1.weight, passes_test: True, RMSE (res-fp64): 0.01335, (ref-fp64): 0.01301 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4280720Z E0605 10:35:02.427000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00658, (ref-fp64): 0.00658 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4286094Z E0605 10:35:02.428000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00820, (ref-fp64): 0.00844 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4291332Z E0605 10:35:02.428000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00875, (ref-fp64): 0.00878 and shape=torch.Size([2304]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4301909Z E0605 10:35:02.429000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01170, (ref-fp64): 0.01173 and shape=torch.Size([2304, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4307120Z E0605 10:35:02.430000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00689, (ref-fp64): 0.00687 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4312584Z E0605 10:35:02.430000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.encoder.layers.9.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00666, (ref-fp64): 0.00668 and shape=torch.Size([768, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4317780Z E0605 10:35:02.431000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.ln_post.bias, passes_test: True, RMSE (res-fp64): 0.00061, (ref-fp64): 0.00053 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4323030Z E0605 10:35:02.431000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.ln_post.weight, passes_test: True, RMSE (res-fp64): 0.00272, (ref-fp64): 0.00259 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4328465Z E0605 10:35:02.432000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.ln_pre.bias, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00920 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4334080Z E0605 10:35:02.432000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.ln_pre.weight, passes_test: True, RMSE (res-fp64): 0.00988, (ref-fp64): 0.00910 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4339307Z E0605 10:35:02.433000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.positional_embedding, passes_test: True, RMSE (res-fp64): 0.01120, (ref-fp64): 0.01107 and shape=torch.Size([50, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4344747Z E0605 10:35:02.434000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_a.projection, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00157 and shape=torch.Size([768, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4350109Z E0605 10:35:02.434000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00669, (ref-fp64): 0.00763 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4357245Z E0605 10:35:02.435000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00783, (ref-fp64): 0.00906 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4362610Z E0605 10:35:02.435000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00584, (ref-fp64): 0.00648 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4369612Z E0605 10:35:02.436000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00695, (ref-fp64): 0.00799 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4374858Z E0605 10:35:02.437000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00625, (ref-fp64): 0.00792 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4380072Z E0605 10:35:02.437000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00703, (ref-fp64): 0.00841 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4385284Z E0605 10:35:02.438000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00643, (ref-fp64): 0.00750 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4390376Z E0605 10:35:02.438000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00789, (ref-fp64): 0.00972 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4395561Z E0605 10:35:02.439000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00919, (ref-fp64): 0.00947 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4401790Z E0605 10:35:02.439000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.00907, (ref-fp64): 0.01007 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4407394Z E0605 10:35:02.440000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00562, (ref-fp64): 0.00695 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4412729Z E0605 10:35:02.440000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.0.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00690, (ref-fp64): 0.00816 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4417714Z E0605 10:35:02.441000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00639, (ref-fp64): 0.00811 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4424832Z E0605 10:35:02.442000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00785, (ref-fp64): 0.00934 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4429561Z E0605 10:35:02.442000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00525, (ref-fp64): 0.00621 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4436499Z E0605 10:35:02.443000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00676, (ref-fp64): 0.00766 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4441328Z E0605 10:35:02.443000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00674, (ref-fp64): 0.00682 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4447053Z E0605 10:35:02.444000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00788, (ref-fp64): 0.00807 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4452268Z E0605 10:35:02.444000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00622, (ref-fp64): 0.00720 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4457579Z E0605 10:35:02.445000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00722, (ref-fp64): 0.00890 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4462528Z E0605 10:35:02.445000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00878, (ref-fp64): 0.00888 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4468496Z E0605 10:35:02.446000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.00878, (ref-fp64): 0.00904 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4473560Z E0605 10:35:02.446000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00593, (ref-fp64): 0.00662 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4478896Z E0605 10:35:02.447000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.1.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00663, (ref-fp64): 0.00763 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4484125Z E0605 10:35:02.447000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00684, (ref-fp64): 0.00779 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4491282Z E0605 10:35:02.448000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00854, (ref-fp64): 0.00941 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4496134Z E0605 10:35:02.449000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00271, (ref-fp64): 0.00282 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4502923Z E0605 10:35:02.449000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00585, (ref-fp64): 0.00611 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4508056Z E0605 10:35:02.450000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00479, (ref-fp64): 0.00540 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4513287Z E0605 10:35:02.450000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00695, (ref-fp64): 0.00706 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4518381Z E0605 10:35:02.451000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00499, (ref-fp64): 0.00546 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4523841Z E0605 10:35:02.451000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00824, (ref-fp64): 0.00904 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4529372Z E0605 10:35:02.452000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00819, (ref-fp64): 0.00817 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4535935Z E0605 10:35:02.453000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01012, (ref-fp64): 0.01024 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4540871Z E0605 10:35:02.453000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00328, (ref-fp64): 0.00398 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4546357Z E0605 10:35:02.454000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.10.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00537, (ref-fp64): 0.00567 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4551454Z E0605 10:35:02.454000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00683, (ref-fp64): 0.00756 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4558296Z E0605 10:35:02.455000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00840, (ref-fp64): 0.00918 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4563411Z E0605 10:35:02.455000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00192, (ref-fp64): 0.00215 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4570612Z E0605 10:35:02.456000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00613, (ref-fp64): 0.00640 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4575709Z E0605 10:35:02.457000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00393, (ref-fp64): 0.00453 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4580873Z E0605 10:35:02.457000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00733, (ref-fp64): 0.00787 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4586049Z E0605 10:35:02.458000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00686, (ref-fp64): 0.00718 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4591121Z E0605 10:35:02.458000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00806, (ref-fp64): 0.00893 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4596286Z E0605 10:35:02.459000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00794, (ref-fp64): 0.00863 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4602260Z E0605 10:35:02.459000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01080, (ref-fp64): 0.01110 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4607794Z E0605 10:35:02.460000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00185, (ref-fp64): 0.00256 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4613331Z E0605 10:35:02.460000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.11.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00520, (ref-fp64): 0.00524 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4618744Z E0605 10:35:02.461000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00672, (ref-fp64): 0.00850 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4625479Z E0605 10:35:02.462000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00811, (ref-fp64): 0.00963 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4630579Z E0605 10:35:02.462000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00445, (ref-fp64): 0.00593 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4637515Z E0605 10:35:02.463000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00633, (ref-fp64): 0.00746 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4642720Z E0605 10:35:02.463000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00549, (ref-fp64): 0.00654 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4648496Z E0605 10:35:02.464000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00668, (ref-fp64): 0.00810 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4653752Z E0605 10:35:02.464000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00742, (ref-fp64): 0.00847 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4658779Z E0605 10:35:02.465000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00779, (ref-fp64): 0.00879 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4664043Z E0605 10:35:02.465000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00869, (ref-fp64): 0.00868 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4670254Z E0605 10:35:02.466000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.00920, (ref-fp64): 0.00947 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4675285Z E0605 10:35:02.467000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00533, (ref-fp64): 0.00630 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4680589Z E0605 10:35:02.467000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.2.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00638, (ref-fp64): 0.00734 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4686089Z E0605 10:35:02.468000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00694, (ref-fp64): 0.00932 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4692812Z E0605 10:35:02.468000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00801, (ref-fp64): 0.01002 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4697968Z E0605 10:35:02.469000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00386, (ref-fp64): 0.00614 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4704715Z E0605 10:35:02.470000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00618, (ref-fp64): 0.00744 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4709732Z E0605 10:35:02.470000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00543, (ref-fp64): 0.00625 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4715049Z E0605 10:35:02.471000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00752, (ref-fp64): 0.00779 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4720050Z E0605 10:35:02.471000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00552, (ref-fp64): 0.00816 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4725455Z E0605 10:35:02.472000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00726, (ref-fp64): 0.00919 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4730785Z E0605 10:35:02.472000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00884, (ref-fp64): 0.00915 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4736859Z E0605 10:35:02.473000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01008, (ref-fp64): 0.01040 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4741721Z E0605 10:35:02.473000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00444, (ref-fp64): 0.00550 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4747161Z E0605 10:35:02.474000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.3.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00615, (ref-fp64): 0.00686 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4752245Z E0605 10:35:02.474000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00706, (ref-fp64): 0.00873 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4758969Z E0605 10:35:02.475000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00804, (ref-fp64): 0.00977 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4763929Z E0605 10:35:02.475000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00440, (ref-fp64): 0.00609 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4771344Z E0605 10:35:02.476000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00609, (ref-fp64): 0.00718 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4776495Z E0605 10:35:02.477000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00652, (ref-fp64): 0.00761 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4781520Z E0605 10:35:02.477000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00772, (ref-fp64): 0.00809 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4786776Z E0605 10:35:02.478000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00552, (ref-fp64): 0.00675 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4791777Z E0605 10:35:02.478000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00642, (ref-fp64): 0.00763 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4797047Z E0605 10:35:02.479000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00857, (ref-fp64): 0.00997 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4803214Z E0605 10:35:02.479000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.00968, (ref-fp64): 0.01080 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4808553Z E0605 10:35:02.480000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00469, (ref-fp64): 0.00624 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4814041Z E0605 10:35:02.480000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.4.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00580, (ref-fp64): 0.00715 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4819034Z E0605 10:35:02.481000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00708, (ref-fp64): 0.00819 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4825945Z E0605 10:35:02.482000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00834, (ref-fp64): 0.00953 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4831031Z E0605 10:35:02.482000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00449, (ref-fp64): 0.00478 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4837940Z E0605 10:35:02.483000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00588, (ref-fp64): 0.00652 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4842952Z E0605 10:35:02.483000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00615, (ref-fp64): 0.00651 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4848432Z E0605 10:35:02.484000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00754, (ref-fp64): 0.00790 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4853515Z E0605 10:35:02.484000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00635, (ref-fp64): 0.00764 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4858499Z E0605 10:35:02.485000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00784, (ref-fp64): 0.00940 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4863830Z E0605 10:35:02.485000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00886, (ref-fp64): 0.00871 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4870061Z E0605 10:35:02.486000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01024, (ref-fp64): 0.01044 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4875427Z E0605 10:35:02.487000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00450, (ref-fp64): 0.00501 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4880573Z E0605 10:35:02.487000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.5.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00562, (ref-fp64): 0.00627 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4885974Z E0605 10:35:02.488000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00674, (ref-fp64): 0.00799 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4892601Z E0605 10:35:02.488000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00814, (ref-fp64): 0.00950 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4897688Z E0605 10:35:02.489000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00416, (ref-fp64): 0.00452 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4904403Z E0605 10:35:02.490000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00579, (ref-fp64): 0.00639 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4909431Z E0605 10:35:02.490000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00585, (ref-fp64): 0.00723 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4914400Z E0605 10:35:02.491000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00759, (ref-fp64): 0.00883 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4919442Z E0605 10:35:02.491000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00611, (ref-fp64): 0.00736 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4924539Z E0605 10:35:02.492000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00722, (ref-fp64): 0.00919 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4929737Z E0605 10:35:02.492000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00843, (ref-fp64): 0.00892 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4935940Z E0605 10:35:02.493000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01030, (ref-fp64): 0.01079 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4941038Z E0605 10:35:02.493000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00420, (ref-fp64): 0.00463 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4946757Z E0605 10:35:02.494000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.6.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00547, (ref-fp64): 0.00573 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4952076Z E0605 10:35:02.494000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00697, (ref-fp64): 0.00819 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4958768Z E0605 10:35:02.495000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00825, (ref-fp64): 0.00958 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4963554Z E0605 10:35:02.495000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00374, (ref-fp64): 0.00427 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4970728Z E0605 10:35:02.496000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00585, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.4975772Z E0605 10:35:02.497000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00617, (ref-fp64): 0.00748 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4980648Z E0605 10:35:02.497000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00788, (ref-fp64): 0.00869 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4985755Z E0605 10:35:02.498000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00619, (ref-fp64): 0.00794 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4990905Z E0605 10:35:02.498000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00764, (ref-fp64): 0.00780 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.4995969Z E0605 10:35:02.499000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00865, (ref-fp64): 0.00880 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5001933Z E0605 10:35:02.499000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01011, (ref-fp64): 0.01047 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5007562Z E0605 10:35:02.500000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00373, (ref-fp64): 0.00418 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5013010Z E0605 10:35:02.500000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.7.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00529, (ref-fp64): 0.00581 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5018057Z E0605 10:35:02.501000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00692, (ref-fp64): 0.00878 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5025015Z E0605 10:35:02.502000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00815, (ref-fp64): 0.00961 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5030090Z E0605 10:35:02.502000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00318, (ref-fp64): 0.00323 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5037027Z E0605 10:35:02.503000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00583, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5042171Z E0605 10:35:02.503000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00544, (ref-fp64): 0.00630 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5048022Z E0605 10:35:02.504000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00690, (ref-fp64): 0.00746 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5052939Z E0605 10:35:02.504000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00477, (ref-fp64): 0.00583 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5058051Z E0605 10:35:02.505000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00709, (ref-fp64): 0.00807 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5063472Z E0605 10:35:02.505000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00781, (ref-fp64): 0.00856 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5069350Z E0605 10:35:02.506000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01008, (ref-fp64): 0.01051 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5074423Z E0605 10:35:02.507000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00329, (ref-fp64): 0.00439 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5079645Z E0605 10:35:02.507000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.8.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00521, (ref-fp64): 0.00583 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5084681Z E0605 10:35:02.508000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.linear1.bias, passes_test: True, RMSE (res-fp64): 0.00654, (ref-fp64): 0.00763 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5091583Z E0605 10:35:02.508000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.linear1.weight, passes_test: True, RMSE (res-fp64): 0.00802, (ref-fp64): 0.00921 and shape=torch.Size([2048, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5096605Z E0605 10:35:02.509000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.linear2.bias, passes_test: True, RMSE (res-fp64): 0.00371, (ref-fp64): 0.00398 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5103815Z E0605 10:35:02.509000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.linear2.weight, passes_test: True, RMSE (res-fp64): 0.00593, (ref-fp64): 0.00641 and shape=torch.Size([512, 2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5108724Z E0605 10:35:02.510000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.norm1.bias, passes_test: True, RMSE (res-fp64): 0.00474, (ref-fp64): 0.00635 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5113881Z E0605 10:35:02.510000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.norm1.weight, passes_test: True, RMSE (res-fp64): 0.00628, (ref-fp64): 0.00749 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5118981Z E0605 10:35:02.511000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.norm2.bias, passes_test: True, RMSE (res-fp64): 0.00527, (ref-fp64): 0.00590 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5124286Z E0605 10:35:02.512000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.norm2.weight, passes_test: True, RMSE (res-fp64): 0.00790, (ref-fp64): 0.00894 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5129810Z E0605 10:35:02.512000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.self_attn.in_proj_bias, passes_test: True, RMSE (res-fp64): 0.00880, (ref-fp64): 0.00907 and shape=torch.Size([1536]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5136078Z E0605 10:35:02.513000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.self_attn.in_proj_weight, passes_test: True, RMSE (res-fp64): 0.01103, (ref-fp64): 0.01121 and shape=torch.Size([1536, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5141317Z E0605 10:35:02.513000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.self_attn.out_proj.bias, passes_test: True, RMSE (res-fp64): 0.00279, (ref-fp64): 0.00327 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5146659Z E0605 10:35:02.514000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.encoder.layers.9.self_attn.out_proj.weight, passes_test: True, RMSE (res-fp64): 0.00542, (ref-fp64): 0.00547 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5151375Z E0605 10:35:02.514000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.ln_final.bias, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00084 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5156487Z E0605 10:35:02.515000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.ln_final.weight, passes_test: True, RMSE (res-fp64): 0.00697, (ref-fp64): 0.00716 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:35:02.5161579Z E0605 10:35:02.515000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.positional_embedding, passes_test: True, RMSE (res-fp64): 0.00145, (ref-fp64): 0.00174 and shape=torch.Size([77, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5167616Z E0605 10:35:02.516000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.projection.weight, passes_test: True, RMSE (res-fp64): 0.00443, (ref-fp64): 0.00467 and shape=torch.Size([512, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5277457Z E0605 10:35:02.527000 140242226967168 torch/_dynamo/utils.py:1482] key: encoder_b.token_embedding.weight, passes_test: True, RMSE (res-fp64): 0.00008, (ref-fp64): 0.00009 and shape=torch.Size([49408, 512]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:02.5315143Z pass
2024-06-05T10:35:02.6115017Z TIMING: entire_frame_compile:100.30911 code_gen:41.96113 inductor_compile:72.34194 backend_compile:86.90345
2024-06-05T10:35:02.6116401Z STATS: call_* op count: 961 | FakeTensor.__torch_dispatch__:26642 | FakeTensorMode.__torch_dispatch__:163037 | ProxyTorchDispatchMode.__torch_dispatch__:40036
2024-06-05T10:35:02.6117592Z Dynamo produced 3 graphs covering 961 ops with 7 graph breaks (5 unique)
2024-06-05T10:35:11.6085676Z 
2024-06-05T10:35:12.2022020Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:35:12.2022597Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:35:12.2023081Z cuda train tts_angular                        
2024-06-05T10:35:15.9883397Z W0605 10:35:15.987000 140566460109440 torch/_logging/_internal.py:1033] [10/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:35:22.9017535Z E0605 10:35:22.900000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.0.linear.weight, passes_test: True, RMSE (res-fp64): 0.00068, (ref-fp64): 0.00070 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9020839Z E0605 10:35:22.901000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.0.lstm.bias_hh_l0, passes_test: True, RMSE (res-fp64): 0.00163, (ref-fp64): 0.00172 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9024591Z E0605 10:35:22.902000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.0.lstm.bias_ih_l0, passes_test: True, RMSE (res-fp64): 0.00163, (ref-fp64): 0.00172 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9037702Z E0605 10:35:22.903000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.0.lstm.weight_hh_l0, passes_test: True, RMSE (res-fp64): 0.00226, (ref-fp64): 0.00226 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9042636Z E0605 10:35:22.903000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.0.lstm.weight_ih_l0, passes_test: True, RMSE (res-fp64): 0.00141, (ref-fp64): 0.00142 and shape=torch.Size([3072, 40]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9048161Z E0605 10:35:22.904000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.1.linear.weight, passes_test: True, RMSE (res-fp64): 0.00069, (ref-fp64): 0.00071 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9052397Z E0605 10:35:22.904000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.1.lstm.bias_hh_l0, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00151 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9056575Z E0605 10:35:22.905000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.1.lstm.bias_ih_l0, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00151 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9069278Z E0605 10:35:22.906000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.1.lstm.weight_hh_l0, passes_test: True, RMSE (res-fp64): 0.00330, (ref-fp64): 0.00330 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9074415Z E0605 10:35:22.907000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.1.lstm.weight_ih_l0, passes_test: True, RMSE (res-fp64): 0.00265, (ref-fp64): 0.00265 and shape=torch.Size([3072, 256]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9079395Z E0605 10:35:22.907000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.2.linear.weight, passes_test: True, RMSE (res-fp64): 0.00056, (ref-fp64): 0.00056 and shape=torch.Size([256, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9083959Z E0605 10:35:22.907000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.2.lstm.bias_hh_l0, passes_test: True, RMSE (res-fp64): 0.00130, (ref-fp64): 0.00135 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9088276Z E0605 10:35:22.908000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.2.lstm.bias_ih_l0, passes_test: True, RMSE (res-fp64): 0.00130, (ref-fp64): 0.00135 and shape=torch.Size([3072]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9100897Z E0605 10:35:22.909000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.2.lstm.weight_hh_l0, passes_test: True, RMSE (res-fp64): 0.00442, (ref-fp64): 0.00441 and shape=torch.Size([3072, 768]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9106220Z E0605 10:35:22.910000 140566460109440 torch/_dynamo/utils.py:1482] key: layers.2.lstm.weight_ih_l0, passes_test: True, RMSE (res-fp64): 0.00342, (ref-fp64): 0.00341 and shape=torch.Size([3072, 256]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:35:22.9110927Z pass
2024-06-05T10:35:22.9177357Z TIMING: entire_frame_compile:8.08166 inductor_compile:6.65728 backend_compile:7.30126 code_gen:5.67247
2024-06-05T10:35:22.9178705Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4838 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:920
2024-06-05T10:35:22.9179891Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique)
2024-06-05T10:35:26.5837252Z 
2024-06-05T10:35:28.9223556Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:35:28.9224152Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:35:28.9224973Z cuda train vgg16                              
2024-06-05T10:36:02.2088785Z E0605 10:36:02.207000 140524185297536 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.00891, (ref-fp64): 0.00945 and shape=torch.Size([4, 1000]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:36:02.2761998Z pass
2024-06-05T10:36:02.2995453Z TIMING: entire_frame_compile:5.85975 code_gen:5.78849 inductor_compile:6.81281 backend_compile:5.5089
2024-06-05T10:36:02.2996783Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:818 | FakeTensorMode.__torch_dispatch__:5985 | ProxyTorchDispatchMode.__torch_dispatch__:1686
2024-06-05T10:36:02.2997949Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique)
2024-06-05T10:36:06.1034261Z 
2024-06-05T10:36:09.2516012Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:36:09.2516788Z loading model: 0it [00:03, ?it/s]
2024-06-05T10:36:09.2517524Z cuda train vision_maskrcnn                    
2024-06-05T10:36:40.2567631Z W0605 10:36:40.256000 140008164496000 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:36:40.2576070Z W0605 10:36:40.257000 140008164496000 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:36:40.2586057Z W0605 10:36:40.258000 140008164496000 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:36:40.2596018Z W0605 10:36:40.259000 140008164496000 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:36:40.2607698Z W0605 10:36:40.260000 140008164496000 torch/_inductor/utils.py:1189] [5/0_1] DeviceCopy in input program
2024-06-05T10:36:42.1703828Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_7). Found from : 
2024-06-05T10:36:42.1705661Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2024-06-05T10:36:42.1706711Z     anchors = self.anchor_generator(images, features)
2024-06-05T10:36:42.1707870Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:36:42.1708810Z     return forward_call(*args, **kwargs)
2024-06-05T10:36:42.1709894Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2024-06-05T10:36:42.1710883Z     self.set_cell_anchors(dtype, device)
2024-06-05T10:36:42.1712006Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2024-06-05T10:36:42.1713346Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-06-05T10:36:42.1714741Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2024-06-05T10:36:42.1716043Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-06-05T10:36:42.1716678Z 
2024-06-05T10:36:43.6198426Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0] Graph break from `Tensor.item()`, consider setting:
2024-06-05T10:36:43.6200826Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-06-05T10:36:43.6202410Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0] or:
2024-06-05T10:36:43.6203538Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-06-05T10:36:43.6204953Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0] to include these operations in the captured graph.
2024-06-05T10:36:43.6206074Z W0605 10:36:43.619000 140008164496000 torch/_dynamo/variables/tensor.py:715] [18/0] 
2024-06-05T10:36:56.0741281Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:36:56.0742948Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:36:56.0744302Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:36:56.0744941Z 
2024-06-05T10:37:02.6726002Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:37:02.6727538Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:37:02.6728747Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:37:02.6729365Z 
2024-06-05T10:37:08.8015443Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:37:08.8016749Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:37:08.8017967Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:37:08.8018614Z 
2024-06-05T10:37:14.4744721Z W0605 10:37:14.473000 140008164496000 torch/fx/experimental/symbolic_shapes.py:4424] [9/1] ps1 is not in var_ranges, defaulting to unknown range.
2024-06-05T10:37:45.9782485Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:37:45.9783781Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:37:45.9784982Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:37:45.9785606Z 
2024-06-05T10:37:55.8243913Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:37:55.8245243Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:37:55.8246691Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:37:55.8247326Z 
2024-06-05T10:38:10.2856326Z W0605 10:38:10.284000 140008164496000 torch/fx/experimental/symbolic_shapes.py:4478] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s3, s4), 0), {s3: evaluate_static_shape_0 + 1, s4: evaluate_static_shape_1 + 1})
2024-06-05T10:38:12.2956754Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:12.2958558Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:38:12.2960322Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:38:12.2961224Z 
2024-06-05T10:38:18.3718858Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:18.3720192Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:38:18.3721389Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:38:18.3722202Z 
2024-06-05T10:38:25.6223207Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:25.6225064Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2024-06-05T10:38:25.6226414Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2024-06-05T10:38:25.6226935Z 
2024-06-05T10:38:26.5490215Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:26.5491841Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2024-06-05T10:38:26.5493213Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2024-06-05T10:38:26.5493736Z 
2024-06-05T10:38:27.6424375Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:27.6426385Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804
2024-06-05T10:38:27.6427633Z     masks_probs = maskrcnn_inference(mask_logits, labels)
2024-06-05T10:38:27.6428862Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference
2024-06-05T10:38:27.6429928Z     mask_prob = mask_prob[index, labels][:, None]
2024-06-05T10:38:27.6430302Z 
2024-06-05T10:38:29.6492711Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:29.6494226Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-06-05T10:38:29.6495409Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-06-05T10:38:29.6496028Z 
2024-06-05T10:38:33.2213567Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:33.2216328Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward
2024-06-05T10:38:33.2217679Z     features = self.backbone(images.tensors)
2024-06-05T10:38:33.2218838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:38:33.2219735Z     return forward_call(*args, **kwargs)
2024-06-05T10:38:33.2220828Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward
2024-06-05T10:38:33.2221785Z     x = self.fpn(x)
2024-06-05T10:38:33.2222651Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
2024-06-05T10:38:33.2223539Z     return forward_call(*args, **kwargs)
2024-06-05T10:38:33.2224586Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward
2024-06-05T10:38:33.2225755Z     inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest")
2024-06-05T10:38:33.2226296Z 
2024-06-05T10:38:38.0146300Z W0605 10:38:38.013000 140008164496000 torch/_dynamo/convert_frame.py:744] [30/8] torch._dynamo hit config.cache_size_limit (8)
2024-06-05T10:38:38.0148268Z W0605 10:38:38.013000 140008164496000 torch/_dynamo/convert_frame.py:744] [30/8]    function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186)
2024-06-05T10:38:38.0150196Z W0605 10:38:38.013000 140008164496000 torch/_dynamo/convert_frame.py:744] [30/8]    last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623
2024-06-05T10:38:38.0151755Z W0605 10:38:38.013000 140008164496000 torch/_dynamo/convert_frame.py:744] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2024-06-05T10:38:38.0153477Z W0605 10:38:38.013000 140008164496000 torch/_dynamo/convert_frame.py:744] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html.
2024-06-05T10:38:46.4042968Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:46.4044543Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:38:46.4045746Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:38:46.4047326Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:38:46.4048275Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:38:46.4049245Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:38:46.4050169Z     return input[
2024-06-05T10:38:46.4050386Z 
2024-06-05T10:38:52.4805106Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:38:52.4807158Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:38:52.4808367Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:38:52.4809707Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:38:52.4810646Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:38:52.4811661Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:38:52.4812520Z     return input[
2024-06-05T10:38:52.4812733Z 
2024-06-05T10:39:00.1518905Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:00.1520268Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:00.1521458Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:00.1522869Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:00.1523832Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:00.1524803Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:00.1525660Z     return input[
2024-06-05T10:39:00.1525884Z 
2024-06-05T10:39:15.1349924Z W0605 10:39:15.133000 140008164496000 torch/fx/experimental/symbolic_shapes.py:4478] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s3: evaluate_static_shape_0 + 1, s4: evaluate_static_shape_1 + 1})
2024-06-05T10:39:16.3767868Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:16.3769336Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:16.3770519Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:16.3771848Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:16.3772789Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:16.3773744Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:16.3774592Z     return input[
2024-06-05T10:39:16.3774808Z 
2024-06-05T10:39:25.7419660Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:25.7421462Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:25.7423160Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:25.7424672Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:25.7425642Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:25.7426949Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:25.7427808Z     return input[
2024-06-05T10:39:25.7428021Z 
2024-06-05T10:39:35.0048280Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:35.0050921Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:35.0053736Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:35.0055115Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:35.0056054Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:35.0057026Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:35.0058114Z     return input[
2024-06-05T10:39:35.0058338Z 
2024-06-05T10:39:43.0486153Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:43.0487940Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:43.0489116Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:43.0490449Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:43.0491391Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:43.0492366Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:43.0493231Z     return input[
2024-06-05T10:39:43.0493445Z 
2024-06-05T10:39:44.1463468Z skipping cudagraphs due to deterministic index put. Found from : 
2024-06-05T10:39:44.1464960Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-06-05T10:39:44.1466181Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-06-05T10:39:44.1467527Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-06-05T10:39:44.1468474Z     v1 = masked_index(y_low, x_low)
2024-06-05T10:39:44.1469439Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-06-05T10:39:44.1470305Z     return input[
2024-06-05T10:39:44.1470523Z 
2024-06-05T10:39:44.3716101Z E0605 10:39:44.370000 140008164496000 torch/_dynamo/utils.py:1482] key: boxes, passes_test: True, RMSE (res-fp64): 0.06520, (ref-fp64): 0.24607 and shape=torch.Size([4, 4]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3725951Z E0605 10:39:44.372000 140008164496000 torch/_dynamo/utils.py:1482] key: masks, passes_test: True, RMSE (res-fp64): 0.00232, (ref-fp64): 0.00435 and shape=torch.Size([4, 1, 427, 640]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3756406Z E0605 10:39:44.375000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00219, (ref-fp64): 0.00267 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3760767Z E0605 10:39:44.375000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00093, (ref-fp64): 0.00123 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3765642Z E0605 10:39:44.376000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00121, (ref-fp64): 0.00157 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3772684Z E0605 10:39:44.376000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00062, (ref-fp64): 0.00072 and shape=torch.Size([512, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3777372Z E0605 10:39:44.377000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00092, (ref-fp64): 0.00098 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3781853Z E0605 10:39:44.377000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00116 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3786406Z E0605 10:39:44.378000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00097, (ref-fp64): 0.00121 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3790887Z E0605 10:39:44.378000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00098, (ref-fp64): 0.00113 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3795083Z E0605 10:39:44.379000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00117 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3799652Z E0605 10:39:44.379000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00116, (ref-fp64): 0.00139 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3804412Z E0605 10:39:44.379000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00139 and shape=torch.Size([128, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3808488Z E0605 10:39:44.380000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00120, (ref-fp64): 0.00146 and shape=torch.Size([128, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3813736Z E0605 10:39:44.380000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer2.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00114, (ref-fp64): 0.00134 and shape=torch.Size([512, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3818106Z E0605 10:39:44.381000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00120, (ref-fp64): 0.00139 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3822462Z E0605 10:39:44.381000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00073, (ref-fp64): 0.00083 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3826811Z E0605 10:39:44.382000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00099, (ref-fp64): 0.00109 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3831295Z E0605 10:39:44.382000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00068, (ref-fp64): 0.00074 and shape=torch.Size([1024, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3835624Z E0605 10:39:44.383000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00061, (ref-fp64): 0.00070 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3840104Z E0605 10:39:44.383000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00076, (ref-fp64): 0.00086 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3845041Z E0605 10:39:44.384000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00107 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3849727Z E0605 10:39:44.384000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00093 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3854615Z E0605 10:39:44.385000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00098 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3858802Z E0605 10:39:44.385000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00106, (ref-fp64): 0.00114 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3863176Z E0605 10:39:44.385000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.3.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00108 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3867421Z E0605 10:39:44.386000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.3.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00100 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3871847Z E0605 10:39:44.386000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.3.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00110 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3876037Z E0605 10:39:44.387000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.4.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00116, (ref-fp64): 0.00121 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3880804Z E0605 10:39:44.387000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.4.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00101, (ref-fp64): 0.00104 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3885100Z E0605 10:39:44.388000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.4.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00101, (ref-fp64): 0.00108 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3889661Z E0605 10:39:44.388000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.5.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00109, (ref-fp64): 0.00115 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3894108Z E0605 10:39:44.388000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.5.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00095, (ref-fp64): 0.00103 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3898497Z E0605 10:39:44.389000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer3.5.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00099, (ref-fp64): 0.00110 and shape=torch.Size([1024, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3902904Z E0605 10:39:44.389000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.0.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00094, (ref-fp64): 0.00102 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3915316Z E0605 10:39:44.391000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.0.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00070 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3922492Z E0605 10:39:44.391000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.0.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00082, (ref-fp64): 0.00094 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3933927Z E0605 10:39:44.392000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.0.downsample.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00086, (ref-fp64): 0.00100 and shape=torch.Size([2048, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3940154Z E0605 10:39:44.393000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.1.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00092 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3952789Z E0605 10:39:44.394000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.1.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00078, (ref-fp64): 0.00089 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3958923Z E0605 10:39:44.395000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.1.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00096, (ref-fp64): 0.00103 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3966360Z E0605 10:39:44.396000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.2.conv1.weight.grad, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00115 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3977842Z E0605 10:39:44.397000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.2.conv2.weight.grad, passes_test: True, RMSE (res-fp64): 0.00080, (ref-fp64): 0.00096 and shape=torch.Size([512, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.3984090Z E0605 10:39:44.397000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.body.layer4.2.conv3.weight.grad, passes_test: True, RMSE (res-fp64): 0.00087, (ref-fp64): 0.00102 and shape=torch.Size([2048, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3988209Z E0605 10:39:44.398000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.0.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00005, (ref-fp64): 0.00077 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3992726Z E0605 10:39:44.398000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00001, (ref-fp64): 0.00017 and shape=torch.Size([256, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.3996338Z E0605 10:39:44.399000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.1.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00005, (ref-fp64): 0.00078 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4001092Z E0605 10:39:44.399000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.1.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00030 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4004839Z E0605 10:39:44.400000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.2.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00186 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4010019Z E0605 10:39:44.400000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00018, (ref-fp64): 0.00035 and shape=torch.Size([256, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4013687Z E0605 10:39:44.400000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.3.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00104, (ref-fp64): 0.00186 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4018556Z E0605 10:39:44.401000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.inner_blocks.3.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00063, (ref-fp64): 0.00093 and shape=torch.Size([256, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4022208Z E0605 10:39:44.401000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.layer_blocks.0.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00057 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4027189Z E0605 10:39:44.402000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.layer_blocks.0.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00003, (ref-fp64): 0.00033 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4034856Z E0605 10:39:44.403000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.layer_blocks.2.0.bias.grad, passes_test: True, RMSE (res-fp64): 0.00079, (ref-fp64): 0.00118 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4039633Z E0605 10:39:44.403000 140008164496000 torch/_dynamo/utils.py:1482] key: backbone.fpn.layer_blocks.2.0.weight.grad, passes_test: True, RMSE (res-fp64): 0.00075, (ref-fp64): 0.00100 and shape=torch.Size([256, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4047812Z E0605 10:39:44.404000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_head.fc6.bias.grad, passes_test: True, RMSE (res-fp64): 0.00036, (ref-fp64): 0.00047 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4104556Z E0605 10:39:44.410000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_head.fc6.weight.grad, passes_test: True, RMSE (res-fp64): 0.00071, (ref-fp64): 0.00083 and shape=torch.Size([1024, 12544]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4108223Z E0605 10:39:44.410000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_head.fc7.bias.grad, passes_test: True, RMSE (res-fp64): 0.00022, (ref-fp64): 0.00066 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4114698Z E0605 10:39:44.411000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_head.fc7.weight.grad, passes_test: True, RMSE (res-fp64): 0.00074, (ref-fp64): 0.00200 and shape=torch.Size([1024, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4118801Z E0605 10:39:44.411000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_predictor.bbox_pred.bias.grad, passes_test: True, RMSE (res-fp64): 0.00005, (ref-fp64): 0.00034 and shape=torch.Size([364]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:39:44.4123310Z E0605 10:39:44.411000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_predictor.bbox_pred.weight.grad, passes_test: True, RMSE (res-fp64): 0.00056, (ref-fp64): 0.00087 and shape=torch.Size([364, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4360673Z E0605 10:39:44.435000 140008164496000 torch/_dynamo/utils.py:1482] key: roi_heads.box_head.fc7.weight, passes_test: True, RMSE (res-fp64): 0.00001, (ref-fp64): 0.00002 and shape=torch.Size([1024, 1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:39:44.4824222Z pass
2024-06-05T10:39:44.5392004Z TIMING: entire_frame_compile:196.99825 inductor_compile:95.75119 backend_compile:155.53584 code_gen:47.68813
2024-06-05T10:39:44.5395959Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:186096 | FakeTensor.__torch_dispatch__:14875 | ProxyTorchDispatchMode.__torch_dispatch__:45365 | attempt fast:3042 | slow no contiguity match:290 | fast is_contiguous:2100 | slow both tensors nontrivially broadcast:652
2024-06-05T10:39:44.5397854Z Dynamo produced 51 graphs covering 2515 ops with 34 graph breaks (8 unique)
2024-06-05T10:39:55.9438428Z 
2024-06-05T10:39:58.7769253Z loading model: 0it [00:00, ?it/s]
2024-06-05T10:39:58.7769767Z loading model: 0it [00:02, ?it/s]
2024-06-05T10:39:58.7770247Z cuda train yolov3                             
2024-06-05T10:40:32.3234998Z W0605 10:40:32.322000 140337053794944 torch/_inductor/utils.py:1189] [9/0] DeviceCopy in input program
2024-06-05T10:40:32.3238394Z W0605 10:40:32.323000 140337053794944 torch/_inductor/utils.py:1189] [9/0] DeviceCopy in input program
2024-06-05T10:40:33.3771101Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_2). Found from : 
2024-06-05T10:40:33.3772574Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-06-05T10:40:33.3773541Z     self.create_grids((nx, ny), p.device)
2024-06-05T10:40:33.3774545Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-06-05T10:40:33.3775573Z     self.anchor_vec = self.anchor_vec.to(device)
2024-06-05T10:40:33.3775940Z 
2024-06-05T10:40:33.7052869Z W0605 10:40:33.704000 140337053794944 torch/_inductor/utils.py:1189] [9/1] DeviceCopy in input program
2024-06-05T10:40:33.7057527Z W0605 10:40:33.705000 140337053794944 torch/_inductor/utils.py:1189] [9/1] DeviceCopy in input program
2024-06-05T10:40:34.9276086Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_4). Found from : 
2024-06-05T10:40:34.9277379Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-06-05T10:40:34.9278337Z     self.create_grids((nx, ny), p.device)
2024-06-05T10:40:34.9279327Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-06-05T10:40:34.9280332Z     self.anchor_vec = self.anchor_vec.to(device)
2024-06-05T10:40:34.9280723Z 
2024-06-05T10:40:35.6853037Z W0605 10:40:35.684000 140337053794944 torch/_logging/_internal.py:1033] [13/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-06-05T10:41:51.6465835Z E0605 10:41:51.645000 140337053794944 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 1.37120, (ref-fp64): 1.37068 and shape=torch.Size([4, 3, 12, 16, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6469952Z E0605 10:41:51.646000 140337053794944 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 1.09269, (ref-fp64): 1.09252 and shape=torch.Size([4, 3, 24, 32, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6482442Z E0605 10:41:51.647000 140337053794944 torch/_dynamo/utils.py:1482] key: , passes_test: True, RMSE (res-fp64): 0.80619, (ref-fp64): 0.80602 and shape=torch.Size([4, 3, 48, 64, 85]). res.dtype: torch.float16, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6490968Z E0605 10:41:51.648000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.0.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.03729, (ref-fp64): 0.03721 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6495109Z E0605 10:41:51.649000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.0.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.04503, (ref-fp64): 0.04563 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6499391Z E0605 10:41:51.649000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.0.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.07061, (ref-fp64): 0.07074 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6503589Z E0605 10:41:51.649000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.01397, (ref-fp64): 0.01371 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6507946Z E0605 10:41:51.650000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01937, (ref-fp64): 0.01929 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6512077Z E0605 10:41:51.650000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.04011, (ref-fp64): 0.04018 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6518831Z E0605 10:41:51.651000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.10.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00610, (ref-fp64): 0.00617 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6523763Z E0605 10:41:51.651000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.10.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01003, (ref-fp64): 0.01008 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6578983Z E0605 10:41:51.657000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00470, (ref-fp64): 0.00471 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6583218Z E0605 10:41:51.657000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00665, (ref-fp64): 0.00670 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6587699Z E0605 10:41:51.658000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00999, (ref-fp64): 0.01004 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6591643Z E0605 10:41:51.658000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00437, (ref-fp64): 0.00439 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6596039Z E0605 10:41:51.659000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00688, (ref-fp64): 0.00686 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6600640Z E0605 10:41:51.659000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01123, (ref-fp64): 0.01130 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6605055Z E0605 10:41:51.660000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.14.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00311, (ref-fp64): 0.00311 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6611855Z E0605 10:41:51.660000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.14.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00639, (ref-fp64): 0.00641 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6618146Z E0605 10:41:51.661000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.16.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00502, (ref-fp64): 0.00501 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6622740Z E0605 10:41:51.661000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.16.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00825, (ref-fp64): 0.00827 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6630999Z E0605 10:41:51.662000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.17.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00448, (ref-fp64): 0.00449 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6635360Z E0605 10:41:51.663000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.19.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00219, (ref-fp64): 0.00216 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6641652Z E0605 10:41:51.663000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.19.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00630, (ref-fp64): 0.00631 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6646129Z E0605 10:41:51.664000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.01536, (ref-fp64): 0.01547 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6650465Z E0605 10:41:51.664000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01682, (ref-fp64): 0.01696 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6654823Z E0605 10:41:51.665000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.03886, (ref-fp64): 0.03901 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6663326Z E0605 10:41:51.665000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.20.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00341, (ref-fp64): 0.00342 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6672063Z E0605 10:41:51.666000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.22.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00518, (ref-fp64): 0.00519 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6680651Z E0605 10:41:51.667000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.23.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00283, (ref-fp64): 0.00284 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6689824Z E0605 10:41:51.668000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.25.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00451, (ref-fp64): 0.00451 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6698411Z E0605 10:41:51.669000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.26.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00244, (ref-fp64): 0.00244 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6707043Z E0605 10:41:51.670000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.28.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00391, (ref-fp64): 0.00392 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6715699Z E0605 10:41:51.671000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.29.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00221, (ref-fp64): 0.00221 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6720169Z E0605 10:41:51.671000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00937, (ref-fp64): 0.00939 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6724479Z E0605 10:41:51.671000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01385, (ref-fp64): 0.01387 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6729665Z E0605 10:41:51.672000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.02507, (ref-fp64): 0.02520 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6737656Z E0605 10:41:51.673000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.31.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00342, (ref-fp64): 0.00344 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6745871Z E0605 10:41:51.674000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.32.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00190, (ref-fp64): 0.00191 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6754304Z E0605 10:41:51.674000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.34.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00324, (ref-fp64): 0.00325 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6763206Z E0605 10:41:51.675000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.35.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00178, (ref-fp64): 0.00178 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6774681Z E0605 10:41:51.676000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.37.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00312, (ref-fp64): 0.00312 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6778828Z E0605 10:41:51.677000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.38.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00185, (ref-fp64): 0.00186 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6785480Z E0605 10:41:51.678000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.38.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00532, (ref-fp64): 0.00531 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6790102Z E0605 10:41:51.678000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.39.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00089, (ref-fp64): 0.00090 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6799232Z E0605 10:41:51.679000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.39.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00241, (ref-fp64): 0.00241 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6808545Z E0605 10:41:51.680000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.41.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00341, (ref-fp64): 0.00340 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6819942Z E0605 10:41:51.681000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.42.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00150, (ref-fp64): 0.00150 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6828329Z E0605 10:41:51.682000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.44.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00252, (ref-fp64): 0.00252 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6839420Z E0605 10:41:51.683000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.45.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00100, (ref-fp64): 0.00100 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6848148Z E0605 10:41:51.684000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.47.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00175, (ref-fp64): 0.00175 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6859522Z E0605 10:41:51.685000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.48.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00068, (ref-fp64): 0.00068 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6863571Z E0605 10:41:51.685000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00935, (ref-fp64): 0.00922 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6867770Z E0605 10:41:51.686000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01290, (ref-fp64): 0.01295 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6872629Z E0605 10:41:51.686000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.02171, (ref-fp64): 0.02178 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6890130Z E0605 10:41:51.688000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.51.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00052, (ref-fp64): 0.00052 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.6898539Z E0605 10:41:51.689000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.53.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00105, (ref-fp64): 0.00104 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6914256Z E0605 10:41:51.690000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.56.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00073, (ref-fp64): 0.00073 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6932130Z E0605 10:41:51.692000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00939, (ref-fp64): 0.00951 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6936381Z E0605 10:41:51.693000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01296, (ref-fp64): 0.01315 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6940726Z E0605 10:41:51.693000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.02330, (ref-fp64): 0.02344 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.6996137Z E0605 10:41:51.698000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.66.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00046, (ref-fp64): 0.00046 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7020979Z E0605 10:41:51.701000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00589, (ref-fp64): 0.00580 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7024973Z E0605 10:41:51.702000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00812, (ref-fp64): 0.00810 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7029669Z E0605 10:41:51.702000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01389, (ref-fp64): 0.01397 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7076471Z E0605 10:41:51.707000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.75.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00078, (ref-fp64): 0.00078 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7102871Z E0605 10:41:51.709000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.76.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00037, (ref-fp64): 0.00037 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7111242Z E0605 10:41:51.710000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.77.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00088, (ref-fp64): 0.00088 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7162894Z E0605 10:41:51.715000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.BatchNorm2d.bias.grad, passes_test: True, RMSE (res-fp64): 0.00605, (ref-fp64): 0.00607 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7167088Z E0605 10:41:51.716000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.BatchNorm2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.00942, (ref-fp64): 0.00953 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7171304Z E0605 10:41:51.716000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.Conv2d.weight.grad, passes_test: True, RMSE (res-fp64): 0.01683, (ref-fp64): 0.01693 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7221155Z E0605 10:41:51.721000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.0.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01463, (ref-fp64): 0.01454 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7227301Z E0605 10:41:51.722000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.0.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01488, (ref-fp64): 0.01490 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7231593Z E0605 10:41:51.722000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01517, (ref-fp64): 0.01521 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7235721Z E0605 10:41:51.723000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01496, (ref-fp64): 0.01496 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7240270Z E0605 10:41:51.723000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.1.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01484 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7244839Z E0605 10:41:51.724000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.10.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01389, (ref-fp64): 0.01411 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7249476Z E0605 10:41:51.724000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.10.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01521, (ref-fp64): 0.01522 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7254492Z E0605 10:41:51.725000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.10.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01493, (ref-fp64): 0.01494 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7262764Z E0605 10:41:51.725000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.103.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01559, (ref-fp64): 0.01555 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7269301Z E0605 10:41:51.726000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.103.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01566, (ref-fp64): 0.01566 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7273884Z E0605 10:41:51.726000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.106.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01746, (ref-fp64): 0.01746 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7278613Z E0605 10:41:51.727000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.106.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01327, (ref-fp64): 0.01329 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7283307Z E0605 10:41:51.727000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.106.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01641, (ref-fp64): 0.01641 and shape=torch.Size([128, 384, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7288269Z E0605 10:41:51.728000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.107.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01571, (ref-fp64): 0.01571 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7292650Z E0605 10:41:51.728000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.107.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01404, (ref-fp64): 0.01412 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7297186Z E0605 10:41:51.729000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.107.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01553, (ref-fp64): 0.01553 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7301314Z E0605 10:41:51.729000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.108.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01601, (ref-fp64): 0.01600 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7307806Z E0605 10:41:51.730000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.108.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01518, (ref-fp64): 0.01518 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7311924Z E0605 10:41:51.730000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.109.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01358, (ref-fp64): 0.01358 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7316243Z E0605 10:41:51.731000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.109.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01405, (ref-fp64): 0.01410 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7320687Z E0605 10:41:51.731000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.109.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01487, (ref-fp64): 0.01487 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7329459Z E0605 10:41:51.732000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.110.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01387, (ref-fp64): 0.01387 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7337925Z E0605 10:41:51.733000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.111.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01340, (ref-fp64): 0.01340 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7345941Z E0605 10:41:51.734000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01474, (ref-fp64): 0.01479 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7350276Z E0605 10:41:51.734000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01505, (ref-fp64): 0.01498 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7354883Z E0605 10:41:51.735000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.12.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7359158Z E0605 10:41:51.735000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01533, (ref-fp64): 0.01501 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7363411Z E0605 10:41:51.735000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01451, (ref-fp64): 0.01471 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7368253Z E0605 10:41:51.736000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.13.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01484 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7372447Z E0605 10:41:51.736000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.14.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01410, (ref-fp64): 0.01419 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7376667Z E0605 10:41:51.737000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.14.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01408, (ref-fp64): 0.01407 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7381571Z E0605 10:41:51.737000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.14.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01481, (ref-fp64): 0.01481 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7385618Z E0605 10:41:51.738000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.16.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01520, (ref-fp64): 0.01500 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7389883Z E0605 10:41:51.738000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.16.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01419, (ref-fp64): 0.01450 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7394353Z E0605 10:41:51.739000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.16.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01466, (ref-fp64): 0.01467 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7398610Z E0605 10:41:51.739000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.17.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01412, (ref-fp64): 0.01409 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7403069Z E0605 10:41:51.739000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.17.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01390, (ref-fp64): 0.01400 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7408057Z E0605 10:41:51.740000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.17.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01482 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7412413Z E0605 10:41:51.740000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.19.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01704, (ref-fp64): 0.01699 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7416668Z E0605 10:41:51.741000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.19.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01489, (ref-fp64): 0.01495 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7421402Z E0605 10:41:51.741000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.19.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01476, (ref-fp64): 0.01475 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7425656Z E0605 10:41:51.742000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01105, (ref-fp64): 0.01103 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7430056Z E0605 10:41:51.742000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01402, (ref-fp64): 0.01414 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7434375Z E0605 10:41:51.743000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.2.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01429, (ref-fp64): 0.01422 and shape=torch.Size([32, 64, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7438617Z E0605 10:41:51.743000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.20.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01490, (ref-fp64): 0.01490 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7443170Z E0605 10:41:51.743000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.20.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01512, (ref-fp64): 0.01511 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7447936Z E0605 10:41:51.744000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.20.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7452270Z E0605 10:41:51.744000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.22.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01499, (ref-fp64): 0.01482 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7456402Z E0605 10:41:51.745000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.22.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01653, (ref-fp64): 0.01639 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7461103Z E0605 10:41:51.745000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.22.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01478, (ref-fp64): 0.01479 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7465309Z E0605 10:41:51.746000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.23.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01448, (ref-fp64): 0.01446 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7469606Z E0605 10:41:51.746000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.23.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01557, (ref-fp64): 0.01554 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7474101Z E0605 10:41:51.746000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.23.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7478311Z E0605 10:41:51.747000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.25.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01357, (ref-fp64): 0.01360 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7482659Z E0605 10:41:51.747000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.25.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01427, (ref-fp64): 0.01401 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7487742Z E0605 10:41:51.748000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.25.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01472, (ref-fp64): 0.01473 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7492105Z E0605 10:41:51.748000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.26.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01482 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7496203Z E0605 10:41:51.749000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.26.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01416, (ref-fp64): 0.01415 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7500790Z E0605 10:41:51.749000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.26.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01477, (ref-fp64): 0.01477 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7505199Z E0605 10:41:51.750000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.28.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01364, (ref-fp64): 0.01372 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7509485Z E0605 10:41:51.750000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.28.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01253, (ref-fp64): 0.01250 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7513993Z E0605 10:41:51.750000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.28.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01484, (ref-fp64): 0.01485 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7518264Z E0605 10:41:51.751000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.29.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01427, (ref-fp64): 0.01428 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7522789Z E0605 10:41:51.751000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.29.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01492 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7527974Z E0605 10:41:51.752000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.29.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7532215Z E0605 10:41:51.752000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01619, (ref-fp64): 0.01622 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7536629Z E0605 10:41:51.753000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01590, (ref-fp64): 0.01593 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7541121Z E0605 10:41:51.753000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.3.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01468, (ref-fp64): 0.01469 and shape=torch.Size([64, 32, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7545639Z E0605 10:41:51.754000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.31.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01483, (ref-fp64): 0.01491 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7549873Z E0605 10:41:51.754000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.31.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01491 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7554588Z E0605 10:41:51.755000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.31.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01537, (ref-fp64): 0.01536 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7558783Z E0605 10:41:51.755000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.32.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01574, (ref-fp64): 0.01567 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7563278Z E0605 10:41:51.755000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.32.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01617, (ref-fp64): 0.01612 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7568377Z E0605 10:41:51.756000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.32.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7572672Z E0605 10:41:51.756000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.34.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01533, (ref-fp64): 0.01534 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7576975Z E0605 10:41:51.757000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.34.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01435, (ref-fp64): 0.01435 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7581486Z E0605 10:41:51.757000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.34.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01566, (ref-fp64): 0.01567 and shape=torch.Size([128, 256, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7585907Z E0605 10:41:51.758000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.35.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.01395 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7590296Z E0605 10:41:51.758000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.35.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01459 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7594924Z E0605 10:41:51.759000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.35.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01481, (ref-fp64): 0.01482 and shape=torch.Size([256, 128, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7599293Z E0605 10:41:51.759000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.37.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01538, (ref-fp64): 0.01528 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7603686Z E0605 10:41:51.759000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.37.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01465, (ref-fp64): 0.01472 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7611110Z E0605 10:41:51.760000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.37.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01484, (ref-fp64): 0.01483 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7615277Z E0605 10:41:51.761000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.38.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01463, (ref-fp64): 0.01463 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7619671Z E0605 10:41:51.761000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.38.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01473, (ref-fp64): 0.01485 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7624243Z E0605 10:41:51.762000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.38.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7628536Z E0605 10:41:51.762000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.39.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01522, (ref-fp64): 0.01518 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7632787Z E0605 10:41:51.762000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.39.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01503, (ref-fp64): 0.01509 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7639967Z E0605 10:41:51.763000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.39.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01477, (ref-fp64): 0.01477 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7644424Z E0605 10:41:51.764000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.41.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01590, (ref-fp64): 0.01588 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7649247Z E0605 10:41:51.764000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.41.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01376, (ref-fp64): 0.01388 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7653775Z E0605 10:41:51.764000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.41.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01466, (ref-fp64): 0.01466 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7657949Z E0605 10:41:51.765000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.42.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01460, (ref-fp64): 0.01452 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7662114Z E0605 10:41:51.765000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.42.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01540, (ref-fp64): 0.01541 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7669336Z E0605 10:41:51.766000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.42.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7673599Z E0605 10:41:51.766000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.44.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01563, (ref-fp64): 0.01547 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7677903Z E0605 10:41:51.767000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.44.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01390, (ref-fp64): 0.01389 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7682741Z E0605 10:41:51.767000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.44.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01475, (ref-fp64): 0.01474 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7687573Z E0605 10:41:51.768000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.45.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01480 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7691936Z E0605 10:41:51.768000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.45.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01449, (ref-fp64): 0.01450 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7699132Z E0605 10:41:51.769000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.45.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01474, (ref-fp64): 0.01474 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7703361Z E0605 10:41:51.769000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.47.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01532, (ref-fp64): 0.01539 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7707609Z E0605 10:41:51.770000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.47.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01419, (ref-fp64): 0.01424 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7712157Z E0605 10:41:51.770000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.47.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01473, (ref-fp64): 0.01474 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7716467Z E0605 10:41:51.771000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.48.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01480, (ref-fp64): 0.01483 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7720663Z E0605 10:41:51.771000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.48.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01595, (ref-fp64): 0.01605 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7728284Z E0605 10:41:51.772000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.48.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01479 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7732572Z E0605 10:41:51.772000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01470, (ref-fp64): 0.01485 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7736699Z E0605 10:41:51.773000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01509, (ref-fp64): 0.01498 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7741707Z E0605 10:41:51.773000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.5.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7745933Z E0605 10:41:51.774000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.50.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01514, (ref-fp64): 0.01513 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7750094Z E0605 10:41:51.774000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.50.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01349, (ref-fp64): 0.01349 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7754880Z E0605 10:41:51.775000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.50.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01470, (ref-fp64): 0.01469 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7758993Z E0605 10:41:51.775000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.51.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01521, (ref-fp64): 0.01529 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7763415Z E0605 10:41:51.775000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.51.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01452, (ref-fp64): 0.01449 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7770964Z E0605 10:41:51.776000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.51.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01481, (ref-fp64): 0.01481 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7775081Z E0605 10:41:51.777000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.53.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01479, (ref-fp64): 0.01473 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7779293Z E0605 10:41:51.777000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.53.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01388, (ref-fp64): 0.01383 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7783890Z E0605 10:41:51.777000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.53.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01490, (ref-fp64): 0.01490 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7788435Z E0605 10:41:51.778000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.54.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01451, (ref-fp64): 0.01446 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7792722Z E0605 10:41:51.778000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.54.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01509, (ref-fp64): 0.01503 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7799953Z E0605 10:41:51.779000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.54.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01485 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7804557Z E0605 10:41:51.780000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.56.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01593, (ref-fp64): 0.01596 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7809274Z E0605 10:41:51.780000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.56.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01397, (ref-fp64): 0.01392 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7814069Z E0605 10:41:51.780000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.56.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01527, (ref-fp64): 0.01527 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7818470Z E0605 10:41:51.781000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.57.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01489, (ref-fp64): 0.01487 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7822832Z E0605 10:41:51.781000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.57.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01469, (ref-fp64): 0.01470 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7829957Z E0605 10:41:51.782000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.57.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7834246Z E0605 10:41:51.783000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.59.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01487, (ref-fp64): 0.01481 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7838411Z E0605 10:41:51.783000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.59.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01503, (ref-fp64): 0.01475 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7843386Z E0605 10:41:51.783000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.59.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01596, (ref-fp64): 0.01596 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7850990Z E0605 10:41:51.784000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01541, (ref-fp64): 0.01584 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7853846Z E0605 10:41:51.784000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01627, (ref-fp64): 0.01619 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7856695Z E0605 10:41:51.785000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.6.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01480, (ref-fp64): 0.01482 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7860863Z E0605 10:41:51.785000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.60.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01493 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7865263Z E0605 10:41:51.786000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.60.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01470, (ref-fp64): 0.01465 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7872291Z E0605 10:41:51.786000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.60.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01478, (ref-fp64): 0.01478 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7876683Z E0605 10:41:51.787000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.62.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01425, (ref-fp64): 0.01429 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7880883Z E0605 10:41:51.787000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.62.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01440, (ref-fp64): 0.01441 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7904087Z E0605 10:41:51.789000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.62.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01483, (ref-fp64): 0.01482 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7908315Z E0605 10:41:51.790000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.63.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01384, (ref-fp64): 0.01382 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7912435Z E0605 10:41:51.790000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.63.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01442, (ref-fp64): 0.01438 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7917241Z E0605 10:41:51.791000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.63.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01461, (ref-fp64): 0.01462 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7921519Z E0605 10:41:51.791000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.64.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01482, (ref-fp64): 0.01478 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7926165Z E0605 10:41:51.792000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.64.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.01390 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7948727Z E0605 10:41:51.794000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.64.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01467, (ref-fp64): 0.01467 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7952758Z E0605 10:41:51.794000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.66.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01519, (ref-fp64): 0.01514 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7957083Z E0605 10:41:51.795000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.66.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01506, (ref-fp64): 0.01504 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7961775Z E0605 10:41:51.795000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.66.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01461, (ref-fp64): 0.01461 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.7966390Z E0605 10:41:51.796000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.67.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01462, (ref-fp64): 0.01460 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7970710Z E0605 10:41:51.796000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.67.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01529, (ref-fp64): 0.01521 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7993210Z E0605 10:41:51.798000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.67.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01463, (ref-fp64): 0.01463 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.7998481Z E0605 10:41:51.799000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.69.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01485, (ref-fp64): 0.01478 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8003470Z E0605 10:41:51.799000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.69.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01456, (ref-fp64): 0.01455 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8007333Z E0605 10:41:51.800000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.69.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01451, (ref-fp64): 0.01451 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8011560Z E0605 10:41:51.800000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01426, (ref-fp64): 0.01421 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8015865Z E0605 10:41:51.801000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01456, (ref-fp64): 0.01456 and shape=torch.Size([128]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8020688Z E0605 10:41:51.801000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.7.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01487, (ref-fp64): 0.01486 and shape=torch.Size([128, 64, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8025221Z E0605 10:41:51.802000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.70.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01405, (ref-fp64): 0.01419 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8029625Z E0605 10:41:51.802000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.70.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01420, (ref-fp64): 0.01426 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8052590Z E0605 10:41:51.804000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.70.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01455, (ref-fp64): 0.01455 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8056583Z E0605 10:41:51.805000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.72.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01455, (ref-fp64): 0.01457 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8061164Z E0605 10:41:51.805000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.72.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01467, (ref-fp64): 0.01461 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8065597Z E0605 10:41:51.806000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.72.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01448, (ref-fp64): 0.01449 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8069826Z E0605 10:41:51.806000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.73.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01431, (ref-fp64): 0.01419 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8074110Z E0605 10:41:51.806000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.73.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01441, (ref-fp64): 0.01447 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8096829Z E0605 10:41:51.809000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.73.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01450, (ref-fp64): 0.01450 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8101065Z E0605 10:41:51.809000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.75.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01347, (ref-fp64): 0.01341 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8105062Z E0605 10:41:51.810000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.75.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01391, (ref-fp64): 0.01384 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8109805Z E0605 10:41:51.810000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.75.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01425, (ref-fp64): 0.01426 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8114084Z E0605 10:41:51.810000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.76.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01385, (ref-fp64): 0.01387 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8118409Z E0605 10:41:51.811000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.76.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01444, (ref-fp64): 0.01444 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8141189Z E0605 10:41:51.813000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.76.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01428, (ref-fp64): 0.01428 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8145495Z E0605 10:41:51.814000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.77.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01386, (ref-fp64): 0.01385 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8149597Z E0605 10:41:51.814000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.77.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01471, (ref-fp64): 0.01473 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8154458Z E0605 10:41:51.815000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.77.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01412, (ref-fp64): 0.01412 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8158681Z E0605 10:41:51.815000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.84.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01260, (ref-fp64): 0.01257 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8167552Z E0605 10:41:51.816000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.84.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01405, (ref-fp64): 0.01405 and shape=torch.Size([512, 2048, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8172004Z E0605 10:41:51.816000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.85.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01361, (ref-fp64): 0.01360 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8176190Z E0605 10:41:51.817000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.85.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01281, (ref-fp64): 0.01281 and shape=torch.Size([1024]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8198671Z E0605 10:41:51.819000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.85.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01359, (ref-fp64): 0.01359 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8203070Z E0605 10:41:51.819000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.86.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01278, (ref-fp64): 0.01280 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8207542Z E0605 10:41:51.820000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.86.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01253, (ref-fp64): 0.01251 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8212295Z E0605 10:41:51.820000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.86.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01321, (ref-fp64): 0.01321 and shape=torch.Size([512, 1024, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8238753Z E0605 10:41:51.823000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.87.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01293, (ref-fp64): 0.01293 and shape=torch.Size([1024, 512, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8247313Z E0605 10:41:51.824000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01368, (ref-fp64): 0.01340 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8251688Z E0605 10:41:51.824000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01531, (ref-fp64): 0.01548 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8255996Z E0605 10:41:51.825000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.9.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01478, (ref-fp64): 0.01479 and shape=torch.Size([64, 128, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8260320Z E0605 10:41:51.825000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.91.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01552, (ref-fp64): 0.01545 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8264636Z E0605 10:41:51.826000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.91.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01455, (ref-fp64): 0.01464 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8269263Z E0605 10:41:51.826000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.91.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01446, (ref-fp64): 0.01446 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8273564Z E0605 10:41:51.826000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.94.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01494, (ref-fp64): 0.01494 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8277696Z E0605 10:41:51.827000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.94.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01395, (ref-fp64): 0.01407 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8282575Z E0605 10:41:51.827000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.94.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01483, (ref-fp64): 0.01483 and shape=torch.Size([256, 768, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8287179Z E0605 10:41:51.828000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.95.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01372, (ref-fp64): 0.01374 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8291812Z E0605 10:41:51.828000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.95.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01398, (ref-fp64): 0.01400 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8298924Z E0605 10:41:51.829000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.95.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01447, (ref-fp64): 0.01447 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8303314Z E0605 10:41:51.829000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.96.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01337, (ref-fp64): 0.01345 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8307662Z E0605 10:41:51.830000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.96.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01328, (ref-fp64): 0.01329 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8312274Z E0605 10:41:51.830000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.96.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01414, (ref-fp64): 0.01414 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8316484Z E0605 10:41:51.831000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.97.BatchNorm2d.bias, passes_test: True, RMSE (res-fp64): 0.01234, (ref-fp64): 0.01244 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8320725Z E0605 10:41:51.831000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.97.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01275, (ref-fp64): 0.01268 and shape=torch.Size([512]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8328426Z E0605 10:41:51.832000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.97.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01421, (ref-fp64): 0.01421 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8334635Z E0605 10:41:51.833000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.98.BatchNorm2d.weight, passes_test: True, RMSE (res-fp64): 0.01307, (ref-fp64): 0.01310 and shape=torch.Size([256]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8339301Z E0605 10:41:51.833000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.98.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01349, (ref-fp64): 0.01349 and shape=torch.Size([256, 512, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000
2024-06-05T10:41:51.8350334Z E0605 10:41:51.834000 140337053794944 torch/_dynamo/utils.py:1482] key: module_list.99.Conv2d.weight, passes_test: True, RMSE (res-fp64): 0.01264, (ref-fp64): 0.01264 and shape=torch.Size([512, 256, 3, 3]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.001000
2024-06-05T10:41:51.8794546Z pass
2024-06-05T10:41:51.9246117Z TIMING: entire_frame_compile:75.15496 inductor_compile:41.89353 backend_compile:58.19917 code_gen:28.19434
2024-06-05T10:41:51.9248088Z STATS: call_* op count: 743 | FakeTensor.__torch_dispatch__:14210 | FakeTensorMode.__torch_dispatch__:78663 | ProxyTorchDispatchMode.__torch_dispatch__:10810 | attempt fast:22 | fast is_contiguous:22
2024-06-05T10:41:51.9249478Z Dynamo produced 31 graphs covering 743 ops with 9 graph breaks (6 unique)
2024-06-05T10:41:55.6978310Z accuracy            pass_rate=76.92%
2024-06-05T10:41:55.6981236Z calls_captured      gmean=0.00x mean=405.564x
2024-06-05T10:41:55.6984335Z unique_graphs       gmean=0.00x mean=4.487x
2024-06-05T10:41:55.6987299Z graph_breaks        gmean=0.00x mean=6.436x
2024-06-05T10:41:55.6990469Z unique_graph_breaks gmean=0.00x mean=4.333x
2024-06-05T10:41:55.6993563Z autograd_captures   gmean=0.00x mean=0.000x
2024-06-05T10:41:55.6997027Z autograd_compiles   gmean=0.00x mean=0.000x
2024-06-05T10:41:55.6999942Z cudagraph_skips     gmean=0.00x mean=0.769x
2024-06-05T10:41:56.8558614Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/cu124/inductor_torchbench_training.csv
2024-06-05T10:41:57.1890058Z lennard_jones                       PASS
2024-06-05T10:41:57.1893915Z llava                               XFAIL
2024-06-05T10:41:57.1898436Z maml_omniglot                       PASS
2024-06-05T10:41:57.1903310Z mnasnet1_0                          PASS
2024-06-05T10:41:57.1908139Z mobilenet_v2                        PASS
2024-06-05T10:41:57.1912901Z mobilenet_v2_quantized_qat          XFAIL
2024-06-05T10:41:57.1917649Z mobilenet_v3_large                  PASS
2024-06-05T10:41:57.1922415Z moco                                PASS
2024-06-05T10:41:57.1927763Z nanogpt                             PASS
2024-06-05T10:41:57.1932286Z nvidia_deeprecommender              PASS
2024-06-05T10:41:57.1937057Z opacus_cifar10                      XFAIL
2024-06-05T10:41:57.1941773Z phlippe_densenet                    PASS
2024-06-05T10:41:57.1946455Z phlippe_resnet                      PASS
2024-06-05T10:41:57.1951256Z pytorch_CycleGAN_and_pix2pix        PASS
2024-06-05T10:41:57.1956135Z pytorch_stargan                     FAIL:     accuracy=fail_to_run, expected=pass
2024-06-05T10:41:57.1960689Z pytorch_unet                        XFAIL
2024-06-05T10:41:57.1965812Z resnet152                           PASS
2024-06-05T10:41:57.1970798Z resnet18                            PASS
2024-06-05T10:41:57.1975535Z resnet50                            PASS
2024-06-05T10:41:57.1980335Z resnet50_quantized_qat              XFAIL
2024-06-05T10:41:57.1985086Z resnext50_32x4d                     PASS
2024-06-05T10:41:57.1989840Z sam                                 XFAIL
2024-06-05T10:41:57.1994689Z shufflenet_v2_x1_0                  PASS
2024-06-05T10:41:57.1999335Z soft_actor_critic                   PASS
2024-06-05T10:41:57.2004455Z speech_transformer                  PASS
2024-06-05T10:41:57.2009500Z squeezenet1_1                       PASS
2024-06-05T10:41:57.2014124Z stable_diffusion_text_encoder       PASS
2024-06-05T10:41:57.2018787Z stable_diffusion_unet               XFAIL
2024-06-05T10:41:57.2023489Z timm_efficientnet                   PASS
2024-06-05T10:41:57.2028226Z timm_regnet                         PASS
2024-06-05T10:41:57.2032991Z timm_resnest                        PASS
2024-06-05T10:41:57.2037644Z timm_vision_transformer             PASS
2024-06-05T10:41:57.2042422Z timm_vision_transformer_large       XFAIL
2024-06-05T10:41:57.2047620Z timm_vovnet                         PASS
2024-06-05T10:41:57.2052264Z torch_multimodal_clip               PASS
2024-06-05T10:41:57.2057031Z tts_angular                         PASS
2024-06-05T10:41:57.2061767Z vgg16                               PASS
2024-06-05T10:41:57.2066550Z vision_maskrcnn                     PASS
2024-06-05T10:41:57.2071382Z yolov3                              PASS
2024-06-05T10:41:57.2071797Z 
2024-06-05T10:41:57.2072110Z Error: 1 models have accuracy status regressed:
2024-06-05T10:41:57.2072942Z     pytorch_stargan
2024-06-05T10:41:57.2073170Z 
2024-06-05T10:41:57.2073175Z 
2024-06-05T10:41:57.2074037Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/cu124/inductor_torchbench_training.csv` to reflect the new baseline.
2024-06-05T10:41:57.2075196Z from pytorch/pytorch root, run
2024-06-05T10:41:57.2076091Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py dffed71f3397e435f3656f25960a4d75ad415746`
2024-06-05T10:41:57.2077430Z and then `git add` the resulting local changes to expected CSVs to your commit.
2024-06-05T10:41:57.2077966Z 
2024-06-05T10:41:57.2555937Z + cleanup_workspace
2024-06-05T10:41:57.2557017Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-06-05T10:41:57.2558181Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-06-05T10:41:57.2559467Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-06-05T10:41:57.2560187Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-06-05T10:41:57.2561039Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-06-05T10:41:57.2561973Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-06-05T10:41:57.2562868Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-06-05T10:41:58.0676460Z ##[error]Process completed with exit code 1.
2024-06-05T10:41:58.0762246Z Prepare all required actions
2024-06-05T10:41:58.0762820Z Getting action download info
2024-06-05T10:41:58.2405286Z ##[group]Run ./.github/actions/pytest-cache-upload
2024-06-05T10:41:58.2405784Z with:
2024-06-05T10:41:58.2406103Z   cache_dir: .pytest_cache
2024-06-05T10:41:58.2406610Z   shard: 2
2024-06-05T10:41:58.2406974Z   sha: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T10:41:58.2407477Z   test_config: inductor_torchbench
2024-06-05T10:41:58.2408048Z   job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T10:41:58.2408611Z env:
2024-06-05T10:41:58.2408912Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:58.2409402Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:58.2410172Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:58.2410839Z ##[endgroup]
2024-06-05T10:41:58.2451537Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-06-05T10:41:58.2452175Z with:
2024-06-05T10:41:58.2452465Z   shell: bash
2024-06-05T10:41:58.2452802Z   timeout_minutes: 5
2024-06-05T10:41:58.2453159Z   max_attempts: 5
2024-06-05T10:41:58.2453511Z   retry_wait_seconds: 30
2024-06-05T10:41:58.2453995Z   command: set -eu
python3 -m pip install boto3==1.19.12

2024-06-05T10:41:58.2454547Z   polling_interval_seconds: 1
2024-06-05T10:41:58.2454955Z   warning_on_retry: true
2024-06-05T10:41:58.2455346Z   continue_on_error: false
2024-06-05T10:41:58.2455712Z env:
2024-06-05T10:41:58.2456019Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:58.2456509Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:58.2457291Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:58.2457952Z ##[endgroup]
2024-06-05T10:41:58.5137556Z Defaulting to user installation because normal site-packages is not writeable
2024-06-05T10:41:58.5296655Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2024-06-05T10:41:58.5351007Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2024-06-05T10:41:58.5401113Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2024-06-05T10:41:58.5429678Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2024-06-05T10:41:58.5444088Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.9.0.post0)
2024-06-05T10:41:58.5471857Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18)
2024-06-05T10:41:58.5669631Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2024-06-05T10:41:59.2959258Z Command completed after 1 attempt(s).
2024-06-05T10:41:59.3001735Z ##[group]Run python3 .github/scripts/pytest_cache.py \
2024-06-05T10:41:59.3002483Z python3 .github/scripts/pytest_cache.py \
2024-06-05T10:41:59.3003009Z   --upload \
2024-06-05T10:41:59.3003481Z   --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \
2024-06-05T10:41:59.3004045Z   --pr_identifier $GITHUB_REF \
2024-06-05T10:41:59.3004597Z   --job_identifier $JOB_IDENTIFIER \
2024-06-05T10:41:59.3005131Z   --sha $SHA \
2024-06-05T10:41:59.3005551Z   --test_config $TEST_CONFIG \
2024-06-05T10:41:59.3006021Z   --shard $SHARD \
2024-06-05T10:41:59.3006554Z   --repo $REPO \
2024-06-05T10:41:59.3006981Z   --temp_dir $RUNNER_TEMP \
2024-06-05T10:41:59.3014911Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.3015578Z env:
2024-06-05T10:41:59.3015901Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.3016406Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.3017185Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.3017885Z   CACHE_DIR: .pytest_cache
2024-06-05T10:41:59.3018420Z   JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-06-05T10:41:59.3019053Z   SHA: dffed71f3397e435f3656f25960a4d75ad415746
2024-06-05T10:41:59.3019558Z   TEST_CONFIG: inductor_torchbench
2024-06-05T10:41:59.3019993Z   SHARD: 2
2024-06-05T10:41:59.3020320Z   REPO: pytorch/pytorch
2024-06-05T10:41:59.3020693Z ##[endgroup]
2024-06-05T10:41:59.5129881Z PR identifier for `refs/tags/ciflow/inductor/127669` is `b0b5e6abbfe2c3236a81c6be5b165415`
2024-06-05T10:41:59.5133159Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/127669', repo='pytorch/pytorch', sha='dffed71f3397e435f3656f25960a4d75ad415746', shard='2', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='inductor_torchbench', upload=True)
2024-06-05T10:41:59.5136259Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload
2024-06-05T10:41:59.5304096Z ##[group]Run cat test/**/*_toprint.log || true
2024-06-05T10:41:59.5304662Z cat test/**/*_toprint.log || true
2024-06-05T10:41:59.5312595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.5313144Z env:
2024-06-05T10:41:59.5313458Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.5313960Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.5314739Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.5315411Z ##[endgroup]
2024-06-05T10:41:59.5382135Z cat: test/**/*_toprint.log: No such file or directory
2024-06-05T10:41:59.5411014Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-06-05T10:41:59.5411525Z kill "$MONITOR_SCRIPT_PID"
2024-06-05T10:41:59.5418328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.5418855Z env:
2024-06-05T10:41:59.5419170Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.5419667Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.5420440Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.5421254Z   MONITOR_SCRIPT_PID: 747
2024-06-05T10:41:59.5421632Z ##[endgroup]
2024-06-05T10:41:59.5615166Z Prepare all required actions
2024-06-05T10:41:59.5615647Z Getting action download info
2024-06-05T10:41:59.6905070Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
2024-06-05T10:41:59.8335420Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-06-05T10:41:59.8335933Z with:
2024-06-05T10:41:59.8336522Z   file-suffix: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532
2024-06-05T10:41:59.8337248Z   s3-bucket: gha-artifacts
2024-06-05T10:41:59.8337635Z env:
2024-06-05T10:41:59.8337943Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.8338447Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.8339232Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.8339903Z ##[endgroup]
2024-06-05T10:41:59.8367344Z ##[group]Run # Remove any previous test jsons if they exist
2024-06-05T10:41:59.8368053Z # Remove any previous test jsons if they exist
2024-06-05T10:41:59.8368607Z rm -f test-jsons-*.zip
2024-06-05T10:41:59.8369170Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2024-06-05T10:41:59.8377118Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.8377661Z env:
2024-06-05T10:41:59.8377978Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.8378470Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.8379256Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.8380212Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532
2024-06-05T10:41:59.8380903Z ##[endgroup]
2024-06-05T10:41:59.8565675Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2024-06-05T10:41:59.8594517Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2024-06-05T10:41:59.8595992Z   adding: test/minioptest_failures_dict.json (deflated 70%)
2024-06-05T10:41:59.8601866Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2024-06-05T10:41:59.8629012Z ##[group]Run # Remove any previous test reports if they exist
2024-06-05T10:41:59.8629705Z # Remove any previous test reports if they exist
2024-06-05T10:41:59.8630273Z rm -f test-reports-*.zip
2024-06-05T10:41:59.8630907Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2024-06-05T10:41:59.8638531Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.8639062Z env:
2024-06-05T10:41:59.8639377Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.8639878Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.8640656Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.8641606Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532
2024-06-05T10:41:59.8642507Z ##[endgroup]
2024-06-05T10:41:59.8813172Z   adding: test/test-reports/inference_torchbench.csv (deflated 69%)
2024-06-05T10:41:59.8815211Z   adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%)
2024-06-05T10:41:59.8816255Z   adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 80%)
2024-06-05T10:41:59.8817176Z   adding: test/test-reports/training_torchbench.csv (deflated 66%)
2024-06-05T10:41:59.8823486Z   adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%)
2024-06-05T10:41:59.8824713Z   adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 80%)
2024-06-05T10:41:59.8851584Z ##[group]Run # Remove any previous usage logs if they exist
2024-06-05T10:41:59.8852256Z # Remove any previous usage logs if they exist
2024-06-05T10:41:59.8852796Z rm -f logs-*.zip
2024-06-05T10:41:59.8853508Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-06-05T10:41:59.8854579Z # so check to see if the file exists first
2024-06-05T10:41:59.8855170Z if [ -f 'usage_log.txt' ]; then
2024-06-05T10:41:59.8855743Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-06-05T10:41:59.8856266Z fi
2024-06-05T10:41:59.8856664Z if ls test/**/*.log 1> /dev/null 2>&1; then
2024-06-05T10:41:59.8857274Z     zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log'
2024-06-05T10:41:59.8857808Z fi
2024-06-05T10:41:59.8864996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.8865529Z env:
2024-06-05T10:41:59.8865832Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.8866322Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.8867092Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.8868027Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532
2024-06-05T10:41:59.8868710Z ##[endgroup]
2024-06-05T10:41:59.8958542Z   adding: usage_log.txt (deflated 92%)
2024-06-05T10:41:59.9031441Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-06-05T10:41:59.9032203Z # Remove any previous debugging artifacts if they exist
2024-06-05T10:41:59.9032789Z rm -f debug-*.zip
2024-06-05T10:41:59.9033204Z if [ -d 'test/debug' ]; then
2024-06-05T10:41:59.9033745Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-06-05T10:41:59.9034258Z fi
2024-06-05T10:41:59.9041185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:41:59.9041724Z env:
2024-06-05T10:41:59.9042023Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.9042672Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.9043443Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.9044377Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532
2024-06-05T10:41:59.9045063Z ##[endgroup]
2024-06-05T10:41:59.9138379Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-06-05T10:41:59.9138854Z with:
2024-06-05T10:41:59.9139166Z   s3-bucket: gha-artifacts
2024-06-05T10:41:59.9139618Z   s3-prefix: pytorch/pytorch/9378671038/1/artifact

2024-06-05T10:41:59.9140122Z   retention-days: 14
2024-06-05T10:41:59.9140489Z   if-no-files-found: warn
2024-06-05T10:41:59.9140879Z   path: test-jsons-*.zip
2024-06-05T10:41:59.9141241Z   name: artifact
2024-06-05T10:41:59.9141568Z   region: us-east-1
2024-06-05T10:41:59.9141896Z env:
2024-06-05T10:41:59.9142189Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:41:59.9142681Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:41:59.9143448Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:41:59.9144111Z ##[endgroup]
2024-06-05T10:42:00.2236039Z NOTE: s3-prefix specified, ignoring name parameter
2024-06-05T10:42:00.2236704Z With the provided path, there will be 1 file uploaded
2024-06-05T10:42:00.2237414Z Uploading to s3 prefix: pytorch/pytorch/9378671038/1/artifact
2024-06-05T10:42:00.2263493Z Starting upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:00.3662138Z Finished upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:00.3809681Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-06-05T10:42:00.3810171Z with:
2024-06-05T10:42:00.3810507Z   s3-bucket: gha-artifacts
2024-06-05T10:42:00.3810982Z   s3-prefix: pytorch/pytorch/9378671038/1/artifact

2024-06-05T10:42:00.3811507Z   retention-days: 14
2024-06-05T10:42:00.3811885Z   if-no-files-found: error
2024-06-05T10:42:00.3812294Z   path: test-reports-*.zip
2024-06-05T10:42:00.3812690Z   name: artifact
2024-06-05T10:42:00.3813037Z   region: us-east-1
2024-06-05T10:42:00.3813372Z env:
2024-06-05T10:42:00.3813687Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:00.3814184Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:00.3815205Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:00.3815880Z ##[endgroup]
2024-06-05T10:42:00.7045695Z NOTE: s3-prefix specified, ignoring name parameter
2024-06-05T10:42:00.7046370Z With the provided path, there will be 1 file uploaded
2024-06-05T10:42:00.7047344Z Uploading to s3 prefix: pytorch/pytorch/9378671038/1/artifact
2024-06-05T10:42:00.7073259Z Starting upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:00.8188957Z Finished upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:00.8343662Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-06-05T10:42:00.8344142Z with:
2024-06-05T10:42:00.8344458Z   s3-bucket: gha-artifacts
2024-06-05T10:42:00.8344951Z   s3-prefix: pytorch/pytorch/9378671038/1/artifact

2024-06-05T10:42:00.8345493Z   retention-days: 14
2024-06-05T10:42:00.8345877Z   if-no-files-found: ignore
2024-06-05T10:42:00.8346282Z   path: logs-*.zip
2024-06-05T10:42:00.8346624Z   name: artifact
2024-06-05T10:42:00.8346953Z   region: us-east-1
2024-06-05T10:42:00.8347287Z env:
2024-06-05T10:42:00.8347594Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:00.8348087Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:00.8348860Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:00.8349534Z ##[endgroup]
2024-06-05T10:42:01.1525397Z NOTE: s3-prefix specified, ignoring name parameter
2024-06-05T10:42:01.1526178Z With the provided path, there will be 1 file uploaded
2024-06-05T10:42:01.1527110Z Uploading to s3 prefix: pytorch/pytorch/9378671038/1/artifact
2024-06-05T10:42:01.1552401Z Starting upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:01.2660333Z Finished upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25823378532.zip
2024-06-05T10:42:01.2806318Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-06-05T10:42:01.2807010Z with:
2024-06-05T10:42:01.2807434Z   s3-bucket: gha-artifacts
2024-06-05T10:42:01.2807909Z   s3-prefix: pytorch/pytorch/9378671038/1/artifact

2024-06-05T10:42:01.2808414Z   retention-days: 14
2024-06-05T10:42:01.2808778Z   if-no-files-found: ignore
2024-06-05T10:42:01.2809173Z   path: debug-*.zip
2024-06-05T10:42:01.2809514Z   name: artifact
2024-06-05T10:42:01.2809838Z   region: us-east-1
2024-06-05T10:42:01.2810165Z env:
2024-06-05T10:42:01.2810470Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:01.2810964Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:01.2811735Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:01.2812408Z ##[endgroup]
2024-06-05T10:42:01.5903994Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-06-05T10:42:01.6052290Z ##[group]Run # shellcheck disable=SC2156
2024-06-05T10:42:01.6052839Z # shellcheck disable=SC2156
2024-06-05T10:42:01.6053685Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-06-05T10:42:01.6061674Z shell: /usr/bin/bash -e {0}
2024-06-05T10:42:01.6062058Z env:
2024-06-05T10:42:01.6062369Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:01.6062867Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:01.6063648Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:01.6064331Z ##[endgroup]
2024-06-05T10:42:01.7735110Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-06-05T10:42:01.7735605Z with:
2024-06-05T10:42:01.7736099Z   name: coredumps-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu
2024-06-05T10:42:01.7736722Z   retention-days: 14
2024-06-05T10:42:01.7737091Z   if-no-files-found: ignore
2024-06-05T10:42:01.7737492Z   path: ./**/core.[1-9]*
2024-06-05T10:42:01.7738014Z   s3-bucket: gha-artifacts
2024-06-05T10:42:01.7738402Z   region: us-east-1
2024-06-05T10:42:01.7738727Z env:
2024-06-05T10:42:01.7739034Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:01.7739524Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:01.7740292Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:01.7740964Z ##[endgroup]
2024-06-05T10:42:11.0211195Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded.
2024-06-05T10:42:11.0445487Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2024-06-05T10:42:11.0446097Z with:
2024-06-05T10:42:11.0446383Z env:
2024-06-05T10:42:11.0446988Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:11.0447487Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:11.0448263Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:11.0448943Z ##[endgroup]
2024-06-05T10:42:11.0484104Z ##[group]Run set -eou pipefail
2024-06-05T10:42:11.0484544Z set -eou pipefail
2024-06-05T10:42:11.0484933Z 
2024-06-05T10:42:11.0485494Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-06-05T10:42:11.0486192Z for _ in $(seq 1440); do
2024-06-05T10:42:11.0486981Z     # Break if no ssh session exists anymore
2024-06-05T10:42:11.0487521Z     if [ "$(who)" = "" ]; then
2024-06-05T10:42:11.0487962Z       break
2024-06-05T10:42:11.0488329Z     fi
2024-06-05T10:42:11.0488662Z     echo "."
2024-06-05T10:42:11.0489022Z     sleep 5
2024-06-05T10:42:11.0489370Z done
2024-06-05T10:42:11.0496964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:42:11.0497508Z env:
2024-06-05T10:42:11.0497820Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:11.0498322Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:11.0499093Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:11.0499784Z ##[endgroup]
2024-06-05T10:42:11.0522849Z Holding runner for 2 hours until all ssh sessions have logged out
2024-06-05T10:42:11.0561196Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-06-05T10:42:11.0562015Z # ignore expansion of "docker ps -q" since it could be empty
2024-06-05T10:42:11.0562873Z # shellcheck disable=SC2046
2024-06-05T10:42:11.0563407Z docker stop $(docker ps -q) || true
2024-06-05T10:42:11.0563932Z # Prune all of the docker images
2024-06-05T10:42:11.0564439Z docker system prune -af
2024-06-05T10:42:11.0572634Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:42:11.0573175Z env:
2024-06-05T10:42:11.0573479Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:11.0573978Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:11.0574755Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:11.0575444Z ##[endgroup]
2024-06-05T10:42:11.4417534Z b23c091d19f2
2024-06-05T10:42:13.8846209Z Deleted Containers:
2024-06-05T10:42:13.8847265Z b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:13.8847833Z 
2024-06-05T10:42:17.7610547Z Deleted Images:
2024-06-05T10:42:17.7612484Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:28a14ba0341ddbf41ea7b800f3d5fd9392fbe0ab
2024-06-05T10:42:17.7615055Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:30399d85d4241c3b46fc1a06bbd1a5a4a40523c847a85377b045014c9718879c
2024-06-05T10:42:17.7616679Z deleted: sha256:4fa961484e1008b77cf438ac442612692dbee4f3476a753574ace973e65df56a
2024-06-05T10:42:17.7617553Z deleted: sha256:030386544d3718cc657d7d3e6a735c419c78a161e6adcc9be23bc10a8b9c8912
2024-06-05T10:42:17.7618684Z deleted: sha256:0d5c29b78436a59643b2355aab1fe9cdfc8808ffe272126958c0d5cd11d149a2
2024-06-05T10:42:17.7619552Z deleted: sha256:adb5cb49e80f329f0381accc7619198a199a848522866f66953f584b59b83dc8
2024-06-05T10:42:17.7620628Z deleted: sha256:c2d8e1c3c7831814dd58c2e348e66ac78036c618ae67113d32e686d79888fc8b
2024-06-05T10:42:17.7621502Z deleted: sha256:26589c25267941b92deecd62b0d38381cf168a44ccac3adafdc85ffb317ebb17
2024-06-05T10:42:17.7622385Z deleted: sha256:6baa0634a6be3726a44d4ff3a509ef5244e5f84e48fdfe9e72100abc76753d34
2024-06-05T10:42:17.7623375Z deleted: sha256:78e414d328bbde6d0a556fe0c676a7f724364926a4fbaca34d6f6689e5bf1bf3
2024-06-05T10:42:17.7624239Z deleted: sha256:9eb3582212b49067ee73cde2b79998cb8ef0315960be70a10190e9d11e25194d
2024-06-05T10:42:17.7625107Z deleted: sha256:8fb2f6ea8ed61e047137fbe3edf48008bc95430594208f88380c329aba36baf5
2024-06-05T10:42:17.7625965Z deleted: sha256:96db213132f0a2e9e3691035f3b7327890604c50614374f6a04a78307ff6418b
2024-06-05T10:42:17.7626840Z deleted: sha256:aa4af9cac4b66d2c4f0b0db161df98fd4d31d5d0228bc2bd7ac67b773504d65c
2024-06-05T10:42:17.7627736Z deleted: sha256:e1e85974e3caa716e9aab7b426dd1149412e0a6c6a2d16b04f806ace0ba6c935
2024-06-05T10:42:17.7628603Z deleted: sha256:a4223efb89f3524536040b69a7814a67b67ffb08da06f3d3dd5ed9ff2ea3d2d2
2024-06-05T10:42:17.7629460Z deleted: sha256:0983748a3b970f882e3c492d7e16aee92b2ac5113d550f1d9272569249b435d3
2024-06-05T10:42:17.7630337Z deleted: sha256:cd3a1d5667bab24c6a0b14684ecb8abf97818edab1710d37dad785bb23372329
2024-06-05T10:42:17.7631225Z deleted: sha256:d6e25c5a6052a3daf6bc839a471f99a8579e75b3c27c6a097e02cd12a5f2afb9
2024-06-05T10:42:17.7632086Z deleted: sha256:2596c666235f934cd14a99ade8ed987c18e6f1ef134174afe309c5f6fdce2ece
2024-06-05T10:42:17.7632958Z deleted: sha256:2309c098cbcdb13a1bfd2d3069a3032540e93e8c17120a6e8be573981732f056
2024-06-05T10:42:17.7633828Z deleted: sha256:18208a26362dab8feb0ee5850f8efa1ad76e96db7775933f78f14534f350e4a3
2024-06-05T10:42:17.7634707Z deleted: sha256:c75b7dd066cdeb90a4257edf19fc9075309c3c1e0b86f9fd3ebce67daa76116b
2024-06-05T10:42:17.7635585Z deleted: sha256:0a52a97602135ae2032b6abdc3bdf185fc2f3ac199f7207d417c662f41755a21
2024-06-05T10:42:17.7636465Z deleted: sha256:c880f8c17d67cbe753abd94eafe6d4673928cebff19b2c5c7723be34bd330d21
2024-06-05T10:42:17.7637346Z deleted: sha256:6455f3bb0b9d9a5540e612e43cc6f318edb27107b0dffa1130c53e7d681da905
2024-06-05T10:42:17.7638216Z deleted: sha256:d66a01d83d750857a5cd1e171bd01178bbb51adc7ebe617bdd8159507e090505
2024-06-05T10:42:17.7639082Z deleted: sha256:737259c79bb3c734d15582466cd71ca5f99b36d7eaa7ca6d1e08b0ce722f3a61
2024-06-05T10:42:17.7639963Z deleted: sha256:c6eaa80e364b8ea951fabf9290469f4c39d24811394c48a4ff3f8be8c6b232f9
2024-06-05T10:42:17.7640840Z deleted: sha256:4bbe060a5b141f56f685b8221699a60d8a2f18571a7a1dc12a9f38387a0bc10e
2024-06-05T10:42:17.7641722Z deleted: sha256:1839c05cb847ae6aabb2e7dec59304137efaba716ebf039f65aaae305614b67c
2024-06-05T10:42:17.7642676Z deleted: sha256:99cafe981c1dffa5116977ce1e701dd007624b14e334e8c2e6b98fce181ca4d8
2024-06-05T10:42:17.7643535Z deleted: sha256:f5f076e9bedee0b002619d563c538f0325623d838175c11d6f3a84318ae3d70a
2024-06-05T10:42:17.7644383Z deleted: sha256:c41801792c5f0079e970ddc9e7b742c8b276a9e93f806d3c4bb889e176977791
2024-06-05T10:42:17.7645237Z deleted: sha256:6418982e15e07b3df4cdc902adbea20304482acbf93822451149fc8edb4a3a49
2024-06-05T10:42:17.7646109Z deleted: sha256:4cba2cb1f84dffdfa8d76670d151b59836bf6cc1b1ee5c0a704b400d819d8c0b
2024-06-05T10:42:17.7647121Z deleted: sha256:9d177f5b3961c68e95520e417ff0a74b3a537f120ba7e0c7f55d61d8a92c84fa
2024-06-05T10:42:17.7647988Z deleted: sha256:88afdfdc3d0603598ba1944c08fe619ec41ac835fc1ce03eb7543d9101d80bbd
2024-06-05T10:42:17.7648858Z deleted: sha256:80f139d3a2ecea893307fab183106bb9da084cebdfc96adb173a2128976d9311
2024-06-05T10:42:17.7649746Z deleted: sha256:bfc9996d7f0f93308d89d6f9f2430cf74a1e9f1cd8b226c3fd78a872dbe2e368
2024-06-05T10:42:17.7650632Z deleted: sha256:e1e7a5d4170c98747ad0d27da4363477cbde88441ec339ac27a464176f849b3f
2024-06-05T10:42:17.7651488Z deleted: sha256:a1aed14bf0f7870143be7099e97240b33261f31dbf9d212dfc3799a1ed0566c6
2024-06-05T10:42:17.7652644Z deleted: sha256:2958191bc6c6cb0c64b8d05d3aa185ca256f0763e8005d1435f38923a8b0f536
2024-06-05T10:42:17.7653502Z deleted: sha256:0955d2ab88c794b31548267c520d606437d042b4a195a3bf943fe26d8da1a65a
2024-06-05T10:42:17.7654338Z deleted: sha256:d351d1e942bfb40a456447b624c48163424442f428f785d3ba1606d835f7cffd
2024-06-05T10:42:17.7655203Z deleted: sha256:bf6fd4775dfbbb27d6816ae1f4148c000ba6e52e6df380ec4f64dc436a48140e
2024-06-05T10:42:17.7656188Z deleted: sha256:bf4eb17ec76a2473342192112d9e9c5601cf7afd1aa6cb63ec3fd667f8d4d57e
2024-06-05T10:42:17.7657067Z deleted: sha256:cf3ef41545419a6a716becf1d02110409b7f41287a6e3b155ccaa1f8ebbcd4f9
2024-06-05T10:42:17.7657924Z deleted: sha256:bd6444db9b8328634c70efd318216dd31e4537e114015332d7c1231ec818c18e
2024-06-05T10:42:17.7658838Z deleted: sha256:44d3659a3ede66c0b91ac06bfb1e16dadcf3c49c651551ea3199ab9923d0b72a
2024-06-05T10:42:17.7659716Z deleted: sha256:b30baf19a931ffdc3ac750b1919377fb9a26388a24c30cd73a479a81b121990c
2024-06-05T10:42:17.7660594Z deleted: sha256:7aa3c0201b05d7de82a3b1d62615f4078d8d69860441f0fed18e7b15c5dfd162
2024-06-05T10:42:17.7661459Z deleted: sha256:922dfa7599f0eb647697333d7f6f604a8172dbfdc8a10a8f8fd4916a0fb457dd
2024-06-05T10:42:17.7662315Z deleted: sha256:102e35998e31a80d86cfb543ec8dea8324a6cb54b8159a8bd49b923bbf645b92
2024-06-05T10:42:17.7663154Z deleted: sha256:604599490b3328e83c752af936b37158664089c1c8384c54435462193a564ebf
2024-06-05T10:42:17.7664015Z deleted: sha256:597be19710e19eaf9dc050ea88cfeafa577bf6a7b6c8ca547165cc39cce4ce5c
2024-06-05T10:42:17.7664888Z deleted: sha256:868f171a555d94f82c5cf1fe8958294e80ea918dae3a61dccb14c7b86e42692b
2024-06-05T10:42:17.7665744Z deleted: sha256:d8672680fa0f5f5cb7cae5d036fed18c2069013f60342fa4c8fd09960b7e19a5
2024-06-05T10:42:17.7666860Z deleted: sha256:687a5be6720a596f1d5cca3c587c107b2bfe445d2974004bda226d308e7507cd
2024-06-05T10:42:17.7667736Z deleted: sha256:0bd88165dfb7e314dfdffefc09f0456c859eee792022e562e54d35e970fbe765
2024-06-05T10:42:17.7668623Z deleted: sha256:cdb83ce3bb6180bf25e5cdc90bc1c06000304e4b4c1f592d517209312110bea9
2024-06-05T10:42:17.7669499Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053
2024-06-05T10:42:17.7670437Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6
2024-06-05T10:42:17.7671318Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737
2024-06-05T10:42:17.7672176Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732
2024-06-05T10:42:17.7673037Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9
2024-06-05T10:42:17.7673905Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650
2024-06-05T10:42:17.7674775Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe
2024-06-05T10:42:17.7675663Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24
2024-06-05T10:42:17.7676554Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c
2024-06-05T10:42:17.7677450Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855
2024-06-05T10:42:17.7678346Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c
2024-06-05T10:42:17.7678880Z 
2024-06-05T10:42:17.7679052Z Total reclaimed space: 42.13GB
2024-06-05T10:42:17.7714788Z ##[group]Run set +e
2024-06-05T10:42:17.7715214Z set +e
2024-06-05T10:42:17.7715542Z set -x
2024-06-05T10:42:17.7715869Z 
2024-06-05T10:42:17.7716182Z nvidia-smi
2024-06-05T10:42:17.7716861Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in
2024-06-05T10:42:17.7717917Z # the case where the driver has already crashed as it still can get the driver version
2024-06-05T10:42:17.7718966Z # and some basic information like the bus ID.  However, the rest of the information
2024-06-05T10:42:17.7719853Z # would be missing (ERR!), for example:
2024-06-05T10:42:17.7720340Z #
2024-06-05T10:42:17.7720791Z # +-----------------------------------------------------------------------------+
2024-06-05T10:42:17.7721615Z # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
2024-06-05T10:42:17.7722582Z # |-------------------------------+----------------------+----------------------+
2024-06-05T10:42:17.7723390Z # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2024-06-05T10:42:17.7724307Z # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2024-06-05T10:42:17.7725103Z # |                               |                      |               MIG M. |
2024-06-05T10:42:17.7725729Z # |===============================+======================+======================|
2024-06-05T10:42:17.7726419Z # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
2024-06-05T10:42:17.7727517Z # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
2024-06-05T10:42:17.7728248Z # |                               |                      |                 ERR! |
2024-06-05T10:42:17.7728910Z # +-------------------------------+----------------------+----------------------+
2024-06-05T10:42:17.7729457Z #
2024-06-05T10:42:17.7729909Z # +-----------------------------------------------------------------------------+
2024-06-05T10:42:17.7730596Z # | Processes:                                                                  |
2024-06-05T10:42:17.7731372Z # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2024-06-05T10:42:17.7732128Z # |        ID   ID                                                   Usage      |
2024-06-05T10:42:17.7732776Z # |=============================================================================|
2024-06-05T10:42:17.7733427Z # +-----------------------------------------------------------------------------+
2024-06-05T10:42:17.7733964Z #
2024-06-05T10:42:17.7734560Z # This should be reported as a failure instead as it will guarantee to fail when
2024-06-05T10:42:17.7735329Z # Docker tries to run with --gpus all
2024-06-05T10:42:17.7735805Z #
2024-06-05T10:42:17.7736376Z # So, the correct check here is to query one of the missing piece of info like
2024-06-05T10:42:17.7737199Z # GPU name, so that the command can fail accordingly
2024-06-05T10:42:17.7737938Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-06-05T10:42:17.7738555Z NVIDIA_SMI_STATUS=$?
2024-06-05T10:42:17.7738946Z 
2024-06-05T10:42:17.7739608Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action
2024-06-05T10:42:17.7740580Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
2024-06-05T10:42:17.7741465Z   echo "NVIDIA driver installation has failed, shutting down the runner..."
2024-06-05T10:42:17.7742223Z   .github/scripts/stop_runner_service.sh
2024-06-05T10:42:17.7742709Z fi
2024-06-05T10:42:17.7743020Z 
2024-06-05T10:42:17.7743742Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the
2024-06-05T10:42:17.7744763Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails
2024-06-05T10:42:17.7745603Z # https://github.com/pytorch/test-infra/issues/4000
2024-06-05T10:42:17.7746247Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l)
2024-06-05T10:42:17.7746767Z NVIDIA_SMI_STATUS=$?
2024-06-05T10:42:17.7747159Z 
2024-06-05T10:42:17.7747810Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action
2024-06-05T10:42:17.7748856Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
2024-06-05T10:42:17.7749734Z   echo "NVIDIA driver installation has failed, shutting down the runner..."
2024-06-05T10:42:17.7750548Z   .github/scripts/stop_runner_service.sh
2024-06-05T10:42:17.7751043Z fi
2024-06-05T10:42:17.7751364Z 
2024-06-05T10:42:17.7751752Z # Check the GPU count to be a power of 2
2024-06-05T10:42:17.7752644Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then
2024-06-05T10:42:17.7753788Z   echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..."
2024-06-05T10:42:17.7754644Z   .github/scripts/stop_runner_service.sh
2024-06-05T10:42:17.7755136Z fi
2024-06-05T10:42:17.7762832Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:42:17.7763371Z env:
2024-06-05T10:42:17.7763672Z   GIT_DEFAULT_BRANCH: main
2024-06-05T10:42:17.7764172Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-06-05T10:42:17.7764947Z   DOCKER_CONTAINER_ID: b23c091d19f24d6632d007d3f087749e74705d04d954ffd0c0e75e5ea81a000e
2024-06-05T10:42:17.7765759Z   RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch
2024-06-05T10:42:17.7766312Z ##[endgroup]
2024-06-05T10:42:17.7786286Z + nvidia-smi
2024-06-05T10:42:19.3238395Z Wed Jun  5 10:42:19 2024       
2024-06-05T10:42:19.3239350Z +-----------------------------------------------------------------------------------------+
2024-06-05T10:42:19.3240233Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-06-05T10:42:19.3241068Z |-----------------------------------------+------------------------+----------------------+
2024-06-05T10:42:19.3241909Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-06-05T10:42:19.3243085Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-06-05T10:42:19.3243828Z |                                         |                        |               MIG M. |
2024-06-05T10:42:19.3244423Z |=========================================+========================+======================|
2024-06-05T10:42:19.3578988Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-06-05T10:42:19.3579838Z |  0%   35C    P0             59W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-06-05T10:42:19.3580519Z |                                         |                        |                  N/A |
2024-06-05T10:42:19.3581341Z +-----------------------------------------+------------------------+----------------------+
2024-06-05T10:42:19.3581959Z                                                                                          
2024-06-05T10:42:19.3582649Z +-----------------------------------------------------------------------------------------+
2024-06-05T10:42:19.3583339Z | Processes:                                                                              |
2024-06-05T10:42:19.3584085Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-06-05T10:42:19.3585248Z |        ID   ID                                                               Usage      |
2024-06-05T10:42:19.3585861Z |=========================================================================================|
2024-06-05T10:42:19.3586556Z |  No running processes found                                                             |
2024-06-05T10:42:19.3587349Z +-----------------------------------------------------------------------------------------+
2024-06-05T10:42:19.9596626Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-06-05T10:42:21.5185957Z NVIDIA A10G
2024-06-05T10:42:21.9537213Z + NVIDIA_SMI_STATUS=0
2024-06-05T10:42:21.9537807Z + '[' 0 -ne 0 ']'
2024-06-05T10:42:21.9541630Z ++ nvidia-smi --list-gpus
2024-06-05T10:42:21.9542055Z ++ wc -l
2024-06-05T10:42:23.9455847Z + GPU_COUNT=1
2024-06-05T10:42:23.9456259Z + NVIDIA_SMI_STATUS=0
2024-06-05T10:42:23.9456816Z + '[' 0 -ne 0 ']'
2024-06-05T10:42:23.9457178Z + '[' 1 -le 8 ']'
2024-06-05T10:42:23.9457540Z + '[' 1 -ne 1 ']'
2024-06-05T10:42:23.9516201Z Post job cleanup.
2024-06-05T10:42:23.9567457Z Post job cleanup.
2024-06-05T10:42:24.0389069Z [command]/usr/bin/git version
2024-06-05T10:42:24.0429769Z git version 2.40.1
2024-06-05T10:42:24.0462040Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/f1e1b595-6dd4-4e62-a7e3-6fb9615add4b/.gitconfig'
2024-06-05T10:42:24.0473108Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f1e1b595-6dd4-4e62-a7e3-6fb9615add4b' before making global git config changes
2024-06-05T10:42:24.0474432Z Adding repository directory to the temporary git global config as a safe directory
2024-06-05T10:42:24.0478018Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-06-05T10:42:24.0511238Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-06-05T10:42:24.0534271Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-06-05T10:42:24.0746468Z Entering 'android/libs/fbjni'
2024-06-05T10:42:24.0782965Z Entering 'third_party/FP16'
2024-06-05T10:42:24.0816976Z Entering 'third_party/FXdiv'
2024-06-05T10:42:24.0851395Z Entering 'third_party/NNPACK'
2024-06-05T10:42:24.0888374Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T10:42:24.0923398Z Entering 'third_party/XNNPACK'
2024-06-05T10:42:24.0967312Z Entering 'third_party/benchmark'
2024-06-05T10:42:24.1001424Z Entering 'third_party/cpp-httplib'
2024-06-05T10:42:24.1037151Z Entering 'third_party/cpuinfo'
2024-06-05T10:42:24.1072531Z Entering 'third_party/cudnn_frontend'
2024-06-05T10:42:24.1106558Z Entering 'third_party/cutlass'
2024-06-05T10:42:24.1149240Z Entering 'third_party/eigen'
2024-06-05T10:42:24.1184173Z Entering 'third_party/fbgemm'
2024-06-05T10:42:24.1218831Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T10:42:24.1253329Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T10:42:24.1287110Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T10:42:24.1327661Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T10:42:24.1361832Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T10:42:24.1396211Z Entering 'third_party/flatbuffers'
2024-06-05T10:42:24.1433816Z Entering 'third_party/fmt'
2024-06-05T10:42:24.1468009Z Entering 'third_party/foxi'
2024-06-05T10:42:24.1502070Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T10:42:24.1538296Z Entering 'third_party/gloo'
2024-06-05T10:42:24.1572680Z Entering 'third_party/googletest'
2024-06-05T10:42:24.1607114Z Entering 'third_party/ideep'
2024-06-05T10:42:24.1641106Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T10:42:24.1681342Z Entering 'third_party/ittapi'
2024-06-05T10:42:24.1718121Z Entering 'third_party/kineto'
2024-06-05T10:42:24.1753988Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T10:42:24.1789134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T10:42:24.1826544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T10:42:24.1862177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T10:42:24.1896538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T10:42:24.1931751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T10:42:24.1967675Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T10:42:24.2001623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T10:42:24.2036708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T10:42:24.2072635Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T10:42:24.2109764Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T10:42:24.2146138Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T10:42:24.2182764Z Entering 'third_party/mimalloc'
2024-06-05T10:42:24.2219258Z Entering 'third_party/nccl/nccl'
2024-06-05T10:42:24.2253743Z Entering 'third_party/nlohmann'
2024-06-05T10:42:24.2287728Z Entering 'third_party/onnx'
2024-06-05T10:42:24.2334422Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T10:42:24.2368778Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T10:42:24.2402061Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T10:42:24.2438219Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T10:42:24.2473384Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T10:42:24.2507475Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T10:42:24.2539876Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T10:42:24.2575636Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T10:42:24.2609662Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T10:42:24.2643129Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T10:42:24.2677319Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T10:42:24.2713781Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T10:42:24.2749567Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T10:42:24.2799560Z Entering 'third_party/pocketfft'
2024-06-05T10:42:24.2836108Z Entering 'third_party/protobuf'
2024-06-05T10:42:24.2873570Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T10:42:24.2906647Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T10:42:24.2943928Z Entering 'third_party/psimd'
2024-06-05T10:42:24.2977885Z Entering 'third_party/pthreadpool'
2024-06-05T10:42:24.3012735Z Entering 'third_party/pybind11'
2024-06-05T10:42:24.3046825Z Entering 'third_party/python-peachpy'
2024-06-05T10:42:24.3080586Z Entering 'third_party/sleef'
2024-06-05T10:42:24.3115007Z Entering 'third_party/tensorpipe'
2024-06-05T10:42:24.3150315Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T10:42:24.3185283Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T10:42:24.3217876Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T10:42:24.3253587Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T10:42:24.3285563Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T10:42:24.3335949Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-06-05T10:42:24.3357019Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3366122Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-06-05T10:42:24.3394070Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-06-05T10:42:24.3602981Z Entering 'android/libs/fbjni'
2024-06-05T10:42:24.3626215Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3651080Z Entering 'third_party/FP16'
2024-06-05T10:42:24.3672310Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3696361Z Entering 'third_party/FXdiv'
2024-06-05T10:42:24.3717828Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3742202Z Entering 'third_party/NNPACK'
2024-06-05T10:42:24.3763925Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3786860Z Entering 'third_party/VulkanMemoryAllocator'
2024-06-05T10:42:24.3808240Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3832082Z Entering 'third_party/XNNPACK'
2024-06-05T10:42:24.3853778Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3887972Z Entering 'third_party/benchmark'
2024-06-05T10:42:24.3909513Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3933879Z Entering 'third_party/cpp-httplib'
2024-06-05T10:42:24.3954647Z http.https://github.com/.extraheader
2024-06-05T10:42:24.3979804Z Entering 'third_party/cpuinfo'
2024-06-05T10:42:24.4000875Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4030111Z Entering 'third_party/cudnn_frontend'
2024-06-05T10:42:24.4048010Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4070060Z Entering 'third_party/cutlass'
2024-06-05T10:42:24.4093427Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4121342Z Entering 'third_party/eigen'
2024-06-05T10:42:24.4143993Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4167862Z Entering 'third_party/fbgemm'
2024-06-05T10:42:24.4188916Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4210854Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-06-05T10:42:24.4231613Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4256328Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-06-05T10:42:24.4277308Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4301688Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-06-05T10:42:24.4323683Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4351292Z Entering 'third_party/fbgemm/third_party/googletest'
2024-06-05T10:42:24.4372216Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4396174Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-06-05T10:42:24.4418494Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4442886Z Entering 'third_party/flatbuffers'
2024-06-05T10:42:24.4466426Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4494046Z Entering 'third_party/fmt'
2024-06-05T10:42:24.4515487Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4539859Z Entering 'third_party/foxi'
2024-06-05T10:42:24.4559873Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4584790Z Entering 'third_party/gemmlowp/gemmlowp'
2024-06-05T10:42:24.4606346Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4628835Z Entering 'third_party/gloo'
2024-06-05T10:42:24.4651377Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4673978Z Entering 'third_party/googletest'
2024-06-05T10:42:24.4696223Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4720057Z Entering 'third_party/ideep'
2024-06-05T10:42:24.4742389Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4765009Z Entering 'third_party/ideep/mkl-dnn'
2024-06-05T10:42:24.4786314Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4815610Z Entering 'third_party/ittapi'
2024-06-05T10:42:24.4837555Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4861674Z Entering 'third_party/kineto'
2024-06-05T10:42:24.4882645Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4906123Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-06-05T10:42:24.4926804Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4951268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-06-05T10:42:24.4972470Z http.https://github.com/.extraheader
2024-06-05T10:42:24.4997033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-06-05T10:42:24.5019282Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5043441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-06-05T10:42:24.5064977Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5089731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-06-05T10:42:24.5110531Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5133860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-06-05T10:42:24.5154777Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5178489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-06-05T10:42:24.5199804Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5223653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-06-05T10:42:24.5244911Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5269006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-06-05T10:42:24.5290656Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5314083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-06-05T10:42:24.5335324Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5359658Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-06-05T10:42:24.5381827Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5406784Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-06-05T10:42:24.5427512Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5453462Z Entering 'third_party/mimalloc'
2024-06-05T10:42:24.5474903Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5496034Z Entering 'third_party/nccl/nccl'
2024-06-05T10:42:24.5517302Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5542584Z Entering 'third_party/nlohmann'
2024-06-05T10:42:24.5563426Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5586674Z Entering 'third_party/onnx'
2024-06-05T10:42:24.5607074Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5643349Z Entering 'third_party/onnx/third_party/benchmark'
2024-06-05T10:42:24.5664597Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5688791Z Entering 'third_party/onnx/third_party/pybind11'
2024-06-05T10:42:24.5710377Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5735317Z Entering 'third_party/opentelemetry-cpp'
2024-06-05T10:42:24.5758182Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5783501Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-06-05T10:42:24.5803757Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5827532Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-06-05T10:42:24.5850452Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5873721Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-06-05T10:42:24.5895601Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5917821Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-06-05T10:42:24.5939475Z http.https://github.com/.extraheader
2024-06-05T10:42:24.5964137Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-06-05T10:42:24.5984700Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6007997Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-06-05T10:42:24.6029265Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6051805Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-06-05T10:42:24.6072696Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6095048Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-06-05T10:42:24.6117021Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6144609Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-06-05T10:42:24.6166819Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6191523Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-06-05T10:42:24.6212910Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6252367Z Entering 'third_party/pocketfft'
2024-06-05T10:42:24.6275100Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6299052Z Entering 'third_party/protobuf'
2024-06-05T10:42:24.6321312Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6346889Z Entering 'third_party/protobuf/third_party/benchmark'
2024-06-05T10:42:24.6368212Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6390649Z Entering 'third_party/protobuf/third_party/googletest'
2024-06-05T10:42:24.6410977Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6436313Z Entering 'third_party/psimd'
2024-06-05T10:42:24.6459666Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6480923Z Entering 'third_party/pthreadpool'
2024-06-05T10:42:24.6503079Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6529536Z Entering 'third_party/pybind11'
2024-06-05T10:42:24.6551507Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6574916Z Entering 'third_party/python-peachpy'
2024-06-05T10:42:24.6596888Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6620142Z Entering 'third_party/sleef'
2024-06-05T10:42:24.6642661Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6667139Z Entering 'third_party/tensorpipe'
2024-06-05T10:42:24.6689174Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6711351Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-06-05T10:42:24.6732663Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6755543Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-06-05T10:42:24.6776905Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6800796Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-06-05T10:42:24.6823374Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6846989Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-06-05T10:42:24.6868030Z http.https://github.com/.extraheader
2024-06-05T10:42:24.6891607Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-06-05T10:42:24.6912531Z http.https://github.com/.extraheader
2024-06-05T10:42:24.7016042Z A job completed hook has been configured by the self-hosted runner administrator
2024-06-05T10:42:24.7034015Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh'
2024-06-05T10:42:24.7040692Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-06-05T10:42:24.7041252Z ##[endgroup]
2024-06-05T10:42:26.0173723Z Cleaning up orphan processes