2023-03-08T02:42:40.3281674Z Requested labels: linux.gcp.a100.large 2023-03-08T02:42:40.3281747Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master 2023-03-08T02:42:40.3281874Z Reusable workflow chain: 2023-03-08T02:42:40.3281903Z pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/master (c88aa336aa0734f42b4d9db7f624d6cfd9b5065e) 2023-03-08T02:42:40.3281943Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master (c88aa336aa0734f42b4d9db7f624d6cfd9b5065e) 2023-03-08T02:42:40.3281968Z Waiting for a runner to pick up this job... 2023-03-08T02:42:40.4944815Z Job is about to start running on the runner: gh-ci-gcp-a100-11 (repository) 2023-03-08T02:42:44.8436471Z Current runner version: '2.302.1' 2023-03-08T02:42:44.8443735Z Runner name: 'gh-ci-gcp-a100-11' 2023-03-08T02:42:44.8444367Z Runner group name: 'Default' 2023-03-08T02:42:44.8445090Z Machine name: 'gh-ci-gcp-a100-11' 2023-03-08T02:42:44.8447485Z ##[group]GITHUB_TOKEN Permissions 2023-03-08T02:42:44.8448385Z Actions: write 2023-03-08T02:42:44.8448764Z Checks: write 2023-03-08T02:42:44.8449063Z Contents: write 2023-03-08T02:42:44.8449474Z Deployments: write 2023-03-08T02:42:44.8449842Z Discussions: write 2023-03-08T02:42:44.8450159Z Issues: write 2023-03-08T02:42:44.8450504Z Metadata: read 2023-03-08T02:42:44.8450857Z Packages: write 2023-03-08T02:42:44.8451195Z Pages: write 2023-03-08T02:42:44.8451555Z PullRequests: write 2023-03-08T02:42:44.8451975Z RepositoryProjects: write 2023-03-08T02:42:44.8452327Z SecurityEvents: write 2023-03-08T02:42:44.8452692Z Statuses: write 2023-03-08T02:42:44.8453065Z ##[endgroup] 2023-03-08T02:42:44.8456668Z Secret source: Actions 2023-03-08T02:42:44.8457351Z Prepare workflow directory 2023-03-08T02:42:45.1155498Z Prepare all required actions 2023-03-08T02:42:45.1387041Z Getting action download info 2023-03-08T02:42:45.3590268Z Download action repository 'pytorch/test-infra@main' (SHA:9f53de2cd40d0132f9a21a03a44ddf0ef92ada70) 2023-03-08T02:42:45.9691152Z Download action repository 'pytorch/pytorch@master' (SHA:c88aa336aa0734f42b4d9db7f624d6cfd9b5065e) 2023-03-08T02:42:50.6226581Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2023-03-08T02:42:51.1763155Z Getting action download info 2023-03-08T02:42:51.3305565Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2023-03-08T02:42:51.7985455Z Getting action download info 2023-03-08T02:42:52.0090174Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2023-03-08T02:42:52.5030740Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master (c88aa336aa0734f42b4d9db7f624d6cfd9b5065e) 2023-03-08T02:42:52.5032672Z ##[group] Inputs 2023-03-08T02:42:52.5033119Z build-environment: linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:42:52.5034001Z test-matrix: { include: [ { config: "inductor_huggingface_perf", shard: 1, num_shards: 1, runner: "linux.gcp.a100.large" }, { config: "inductor_timm_perf", shard: 1, num_shards: 2, runner: "linux.gcp.a100.large" }, { config: "inductor_timm_perf", shard: 2, num_shards: 2, runner: "linux.gcp.a100.large" }, { config: "inductor_torchbench_perf", shard: 1, num_shards: 1, runner: "linux.gcp.a100.large" }, ]} 2023-03-08T02:42:52.5034906Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:42:52.5035354Z sync-tag: 2023-03-08T02:42:52.5036182Z timeout-minutes: 1200 2023-03-08T02:42:52.5036479Z use-gha: anything-non-empty-to-use-gha 2023-03-08T02:42:52.5036767Z ##[endgroup] 2023-03-08T02:42:52.5037365Z Complete job name: cuda11.8-py3.10-gcc7-sm80 / test (inductor_huggingface_perf, 1, 1, linux.gcp.a100.large) 2023-03-08T02:42:52.5894798Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2023-03-08T02:42:52.5895146Z with: 2023-03-08T02:42:52.5895740Z github-secret: *** 2023-03-08T02:42:52.5896156Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-03-08T02:42:52.5896644Z activate-with-label: false 2023-03-08T02:42:52.5896891Z label: with-ssh 2023-03-08T02:42:52.5897122Z remove-existing-keys: true 2023-03-08T02:42:52.5897357Z env: 2023-03-08T02:42:52.5897575Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:42:52.5897794Z ##[endgroup] 2023-03-08T02:42:52.6857729Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2023-03-08T02:42:52.7118630Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2023-03-08T02:42:52.7119087Z with: 2023-03-08T02:42:52.7119294Z submodules: recursive 2023-03-08T02:42:52.7119544Z fetch-depth: 0 2023-03-08T02:42:52.7119795Z env: 2023-03-08T02:42:52.7119995Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:42:52.7120226Z ##[endgroup] 2023-03-08T02:42:52.7369721Z ##[group]Run retry () { 2023-03-08T02:42:52.7370033Z retry () { 2023-03-08T02:42:52.7370317Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2023-03-08T02:42:52.7370570Z } 2023-03-08T02:42:52.7370805Z echo "${GITHUB_WORKSPACE}" 2023-03-08T02:42:52.7371073Z if [ -z "${NO_SUDO}" ]; then 2023-03-08T02:42:52.7371341Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2023-03-08T02:42:52.7371593Z else 2023-03-08T02:42:52.7371837Z  retry rm -rf "${GITHUB_WORKSPACE}" 2023-03-08T02:42:52.7372064Z fi 2023-03-08T02:42:52.7372342Z mkdir "${GITHUB_WORKSPACE}" 2023-03-08T02:42:52.7395034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:42:52.7395340Z env: 2023-03-08T02:42:52.7395567Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:42:52.7395803Z NO_SUDO: 2023-03-08T02:42:52.7396004Z ##[endgroup] 2023-03-08T02:42:52.7515429Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T02:42:55.5351380Z ##[group]Run malfet/checkout@silent-checkout 2023-03-08T02:42:55.5351702Z with: 2023-03-08T02:42:55.5351956Z ref: c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:42:55.5352204Z fetch-depth: 0 2023-03-08T02:42:55.5352428Z submodules: recursive 2023-03-08T02:42:55.5352659Z quiet-checkout: true 2023-03-08T02:42:55.5352892Z repository: pytorch/pytorch 2023-03-08T02:42:55.5353349Z token: *** 2023-03-08T02:42:55.5353562Z ssh-strict: true 2023-03-08T02:42:55.5353802Z persist-credentials: true 2023-03-08T02:42:55.5354025Z clean: true 2023-03-08T02:42:55.5354233Z lfs: false 2023-03-08T02:42:55.5354459Z set-safe-directory: true 2023-03-08T02:42:55.5354668Z env: 2023-03-08T02:42:55.5354891Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:42:55.5355115Z ##[endgroup] 2023-03-08T02:42:55.6703062Z Syncing repository: pytorch/pytorch 2023-03-08T02:42:55.6704985Z ##[group]Getting Git version info 2023-03-08T02:42:55.6705557Z Working directory is '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch' 2023-03-08T02:42:55.6706302Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git' 2023-03-08T02:42:55.6707279Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git' 2023-03-08T02:42:55.6707960Z [command]/usr/bin/git version 2023-03-08T02:42:55.6708210Z git version 2.25.1 2023-03-08T02:42:55.6712925Z ##[endgroup] 2023-03-08T02:42:55.6726502Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/0b933889-223f-4db6-9818-2f7cd0a690b5' before making global git config changes 2023-03-08T02:42:55.6727141Z Adding repository directory to the temporary git global config as a safe directory 2023-03-08T02:42:55.6731429Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T02:42:55.6776060Z Deleting the contents of '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch' 2023-03-08T02:42:55.6781742Z ##[group]Initializing the repository 2023-03-08T02:42:55.6784955Z [command]/usr/bin/git init /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T02:42:55.6827262Z Initialized empty Git repository in /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/ 2023-03-08T02:42:55.6835846Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2023-03-08T02:42:55.6877271Z ##[endgroup] 2023-03-08T02:42:55.6877965Z ##[group]Disabling automatic garbage collection 2023-03-08T02:42:55.6880879Z [command]/usr/bin/git config --local gc.auto 0 2023-03-08T02:42:55.6913735Z ##[endgroup] 2023-03-08T02:42:55.6914159Z ##[group]Setting up auth 2023-03-08T02:42:55.6921481Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-03-08T02:42:55.6957405Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2023-03-08T02:42:55.7204885Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-03-08T02:42:55.7237360Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2023-03-08T02:42:55.7482542Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-03-08T02:42:55.7523463Z ##[endgroup] 2023-03-08T02:42:55.7523921Z ##[group]Fetching the repository 2023-03-08T02:42:55.7530971Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2023-03-08T02:44:06.7636479Z [command]/usr/bin/git rev-parse --verify --quiet c88aa336aa0734f42b4d9db7f624d6cfd9b5065e^{object} 2023-03-08T02:44:06.7666238Z c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:44:06.7675401Z ##[endgroup] 2023-03-08T02:44:06.7676076Z ##[group]Determining the checkout info 2023-03-08T02:44:06.7676502Z ##[endgroup] 2023-03-08T02:44:06.7676909Z ##[group]Checking out the ref 2023-03-08T02:44:06.7677553Z [command]/usr/bin/git checkout --quiet --force c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:44:08.3243700Z ##[endgroup] 2023-03-08T02:44:08.3244227Z ##[group]Setting up auth for fetching submodules 2023-03-08T02:44:08.3247739Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-03-08T02:44:08.3300043Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2023-03-08T02:44:08.3334675Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2023-03-08T02:44:08.3370591Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2023-03-08T02:44:08.3401000Z ##[endgroup] 2023-03-08T02:44:08.3401443Z ##[group]Fetching submodules 2023-03-08T02:44:08.3406203Z [command]/usr/bin/git submodule sync --recursive 2023-03-08T02:44:08.3672244Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2023-03-08T02:44:08.3933093Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2023-03-08T02:44:08.3938480Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2023-03-08T02:44:08.3945193Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2023-03-08T02:44:08.3952587Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2023-03-08T02:44:08.3959244Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2023-03-08T02:44:08.3966029Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2023-03-08T02:44:08.3972381Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2023-03-08T02:44:08.3978943Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2023-03-08T02:44:08.3985849Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2023-03-08T02:44:08.3993349Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2023-03-08T02:44:08.3999423Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2023-03-08T02:44:08.4006470Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2023-03-08T02:44:08.4013126Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2023-03-08T02:44:08.4020273Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2023-03-08T02:44:08.4027063Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2023-03-08T02:44:08.4033815Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2023-03-08T02:44:08.4040849Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2023-03-08T02:44:08.4047707Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:44:08.4054949Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2023-03-08T02:44:08.4061776Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2023-03-08T02:44:08.4069722Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2023-03-08T02:44:08.4077348Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2023-03-08T02:44:08.4084221Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2023-03-08T02:44:08.4092007Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2023-03-08T02:44:08.4098383Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2023-03-08T02:44:08.4105520Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2023-03-08T02:44:08.4112968Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2023-03-08T02:44:08.4119858Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2023-03-08T02:44:08.4127366Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2023-03-08T02:44:08.4134881Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2023-03-08T02:44:08.4142219Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2023-03-08T02:44:08.4149832Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2023-03-08T02:44:08.4157751Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2023-03-08T02:44:08.4165118Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2023-03-08T02:44:08.4173681Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2023-03-08T02:44:08.4181621Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2023-03-08T02:44:08.4189336Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2023-03-08T02:44:08.4197173Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2023-03-08T02:44:08.4204994Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2023-03-08T02:44:08.4213416Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2023-03-08T02:44:08.4221081Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2023-03-08T02:44:08.4293334Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2023-03-08T02:44:08.9543152Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2023-03-08T02:44:09.3585956Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2023-03-08T02:44:09.8976234Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2023-03-08T02:44:10.4185789Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2023-03-08T02:44:11.0144913Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2023-03-08T02:44:14.1731930Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2023-03-08T02:44:22.9396964Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2023-03-08T02:44:23.5914376Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2023-03-08T02:44:24.3967635Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2023-03-08T02:44:26.2538744Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2023-03-08T02:44:27.8627673Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2023-03-08T02:44:30.1534711Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2023-03-08T02:44:35.4851708Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2023-03-08T02:44:36.5533804Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2023-03-08T02:44:38.2901410Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2023-03-08T02:44:40.0330601Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2023-03-08T02:44:40.4200084Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2023-03-08T02:44:41.1637809Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2023-03-08T02:44:41.7262260Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2023-03-08T02:44:43.2841912Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2023-03-08T02:44:44.0616401Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2023-03-08T02:44:44.4395034Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2023-03-08T02:44:44.9361970Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2023-03-08T02:44:46.6159441Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2023-03-08T02:44:47.2679598Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2023-03-08T02:44:47.8520976Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2023-03-08T02:44:54.9756288Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2023-03-08T02:44:57.4268838Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2023-03-08T02:44:58.1510603Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2023-03-08T02:44:58.6102292Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2023-03-08T02:45:09.4050229Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2023-03-08T02:45:09.7867935Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2023-03-08T02:45:10.2796216Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2023-03-08T02:45:11.4874372Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2023-03-08T02:45:11.8966867Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2023-03-08T02:45:12.6452527Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2023-03-08T02:45:13.1956522Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2023-03-08T02:45:14.0587769Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2023-03-08T02:45:16.6614192Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2023-03-08T02:45:17.3951512Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2023-03-08T02:45:20.2683029Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2023-03-08T02:45:20.3042734Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2023-03-08T02:45:20.3378123Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2023-03-08T02:45:20.3864011Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2023-03-08T02:45:20.4348558Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2023-03-08T02:45:20.5010347Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2023-03-08T02:45:21.2407442Z Submodule path 'third_party/XNNPACK': checked out '51a987591a6fc9f0fc0707077f53d763ac132cbf' 2023-03-08T02:45:21.2894905Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-03-08T02:45:21.4279769Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2023-03-08T02:45:21.4900808Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2023-03-08T02:45:21.8228774Z Submodule path 'third_party/cudnn_frontend': checked out '81a041a68245cd8f871c43bbbbd5b6b627979a30' 2023-03-08T02:45:22.2823061Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2023-03-08T02:45:22.5762866Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2023-03-08T02:45:22.6540277Z Submodule path 'third_party/fbgemm': checked out '03b2046676707da64504e898490ab46104d4682a' 2023-03-08T02:45:22.6590135Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:45:22.6597585Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:45:22.6604660Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:45:22.6610058Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:45:22.6616745Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:45:22.6662103Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2023-03-08T02:45:23.6485791Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2023-03-08T02:45:24.4337692Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2023-03-08T02:45:26.6107733Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2023-03-08T02:45:28.2610448Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2023-03-08T02:45:28.8276032Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2023-03-08T02:45:28.9636542Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2023-03-08T02:45:29.4207590Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2023-03-08T02:45:29.5091770Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2023-03-08T02:45:29.5420433Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2023-03-08T02:45:29.6601121Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2023-03-08T02:45:29.7221888Z Submodule path 'third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2023-03-08T02:45:29.7576173Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2023-03-08T02:45:29.8252146Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2023-03-08T02:45:29.8744084Z Submodule path 'third_party/gloo': checked out '10909297fedab0a680799211a299203e53515032' 2023-03-08T02:45:29.9495724Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2023-03-08T02:45:29.9871303Z Submodule path 'third_party/ideep': checked out '7bc3e12f7c0cad7fb24f8d4ab63dcd467ffa60c7' 2023-03-08T02:45:29.9920361Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2023-03-08T02:45:29.9964120Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2023-03-08T02:45:39.6739708Z Submodule path 'third_party/ideep/mkl-dnn': checked out '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e' 2023-03-08T02:45:39.7105562Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2023-03-08T02:45:39.7512823Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2023-03-08T02:45:39.8749085Z Submodule path 'third_party/kineto': checked out '9380d64055137e609709b4b72230143848ca3465' 2023-03-08T02:45:39.8798857Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:45:39.8803938Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:45:39.8811730Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:45:39.8858327Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2023-03-08T02:45:40.5541765Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2023-03-08T02:45:42.2359731Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2023-03-08T02:45:43.8807869Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2023-03-08T02:45:43.8858284Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:45:43.8863536Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:45:43.8870104Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:45:43.8876825Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:45:43.8883121Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:45:43.8889457Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:45:43.8895823Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:45:43.8901554Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:45:43.8948596Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2023-03-08T02:45:44.8838594Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2023-03-08T02:45:45.4933169Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2023-03-08T02:45:47.1837553Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2023-03-08T02:45:47.8072091Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2023-03-08T02:45:48.7191896Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2023-03-08T02:45:50.2725318Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2023-03-08T02:45:57.5116207Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2023-03-08T02:45:58.1871868Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2023-03-08T02:45:58.2268901Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2023-03-08T02:45:58.2892785Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2023-03-08T02:45:58.3247913Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2023-03-08T02:45:58.3296138Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:45:58.3340323Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2023-03-08T02:45:59.0065059Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2023-03-08T02:45:59.0455631Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2023-03-08T02:45:59.1116116Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2023-03-08T02:45:59.2441774Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2023-03-08T02:45:59.2824677Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2023-03-08T02:45:59.3404478Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2023-03-08T02:45:59.4218600Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2023-03-08T02:45:59.4695314Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2023-03-08T02:45:59.5086798Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2023-03-08T02:45:59.6466762Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2023-03-08T02:45:59.9972183Z Submodule path 'third_party/onnx': checked out 'e192ba01e438d22ca2dedd7956e28e3551626c91' 2023-03-08T02:46:00.0036606Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:00.0041757Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:00.0104742Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2023-03-08T02:46:00.6977725Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2023-03-08T02:46:01.9282701Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-03-08T02:46:01.9914878Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '914c06fb252b6cc3727d0eedab6736e88a3fcb01' 2023-03-08T02:46:02.0336681Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2023-03-08T02:46:02.0490400Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:02.0533857Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2023-03-08T02:46:04.7169126Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2023-03-08T02:46:04.7223968Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:04.7231190Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:04.7282900Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2023-03-08T02:46:05.4006369Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2023-03-08T02:46:06.6050358Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2023-03-08T02:46:06.7022930Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2023-03-08T02:46:06.7067955Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:06.7113558Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2023-03-08T02:46:07.1785410Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-03-08T02:46:07.2136929Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2023-03-08T02:46:07.5210277Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2023-03-08T02:46:07.5267083Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:07.5273465Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:07.5323715Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2023-03-08T02:46:08.2466711Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2023-03-08T02:46:09.8105850Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2023-03-08T02:46:09.9079920Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2023-03-08T02:46:09.9414159Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2023-03-08T02:46:09.9776750Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2023-03-08T02:46:10.0369596Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2023-03-08T02:46:10.0719073Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2023-03-08T02:46:10.1277192Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2023-03-08T02:46:10.1608939Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2023-03-08T02:46:10.2329669Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2023-03-08T02:46:10.3776002Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2023-03-08T02:46:10.4292492Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2023-03-08T02:46:10.4341777Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:10.4347329Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:10.4352725Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:10.4358625Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:10.4402501Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2023-03-08T02:46:11.9923674Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2023-03-08T02:46:12.5537497Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2023-03-08T02:46:14.0411831Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2023-03-08T02:46:15.2906744Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2023-03-08T02:46:15.3269978Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2023-03-08T02:46:15.4177888Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2023-03-08T02:46:15.4689314Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2023-03-08T02:46:15.4738233Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:15.4782769Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2023-03-08T02:46:15.9438335Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-03-08T02:46:16.1170140Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2023-03-08T02:46:16.1253639Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2023-03-08T02:46:16.1522402Z Entering 'android/libs/fbjni' 2023-03-08T02:46:16.1560944Z Entering 'third_party/FP16' 2023-03-08T02:46:16.1599959Z Entering 'third_party/FXdiv' 2023-03-08T02:46:16.1639385Z Entering 'third_party/NNPACK' 2023-03-08T02:46:16.1678678Z Entering 'third_party/QNNPACK' 2023-03-08T02:46:16.1718702Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T02:46:16.1759534Z Entering 'third_party/XNNPACK' 2023-03-08T02:46:16.1813036Z Entering 'third_party/benchmark' 2023-03-08T02:46:16.1852693Z Entering 'third_party/cpuinfo' 2023-03-08T02:46:16.1893988Z Entering 'third_party/cub' 2023-03-08T02:46:16.1932752Z Entering 'third_party/cudnn_frontend' 2023-03-08T02:46:16.1978631Z Entering 'third_party/cutlass' 2023-03-08T02:46:16.2024164Z Entering 'third_party/eigen' 2023-03-08T02:46:16.2065887Z Entering 'third_party/fbgemm' 2023-03-08T02:46:16.2105192Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:46:16.2142754Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:46:16.2180010Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:46:16.2224923Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:46:16.2261876Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:46:16.2300187Z Entering 'third_party/flatbuffers' 2023-03-08T02:46:16.2340490Z Entering 'third_party/fmt' 2023-03-08T02:46:16.2379541Z Entering 'third_party/foxi' 2023-03-08T02:46:16.2417115Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:46:16.2456116Z Entering 'third_party/gloo' 2023-03-08T02:46:16.2494154Z Entering 'third_party/googletest' 2023-03-08T02:46:16.2533028Z Entering 'third_party/ideep' 2023-03-08T02:46:16.2570807Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T02:46:16.2616607Z Entering 'third_party/ios-cmake' 2023-03-08T02:46:16.2654812Z Entering 'third_party/ittapi' 2023-03-08T02:46:16.2692886Z Entering 'third_party/kineto' 2023-03-08T02:46:16.2731627Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:46:16.2769115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:46:16.2807287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:46:16.2844369Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:46:16.2883275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:46:16.2919879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:46:16.2967232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:46:16.3003107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:46:16.3041435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:46:16.3080428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:46:16.3120064Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:46:16.3157553Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:46:16.3196872Z Entering 'third_party/nccl/nccl' 2023-03-08T02:46:16.3235728Z Entering 'third_party/neon2sse' 2023-03-08T02:46:16.3274378Z Entering 'third_party/nlohmann' 2023-03-08T02:46:16.3314738Z Entering 'third_party/onnx' 2023-03-08T02:46:16.3366542Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:16.3404071Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:16.3445342Z Entering 'third_party/onnx-tensorrt' 2023-03-08T02:46:16.3483199Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:16.3525458Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:16.3563796Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:16.3601693Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:16.3644953Z Entering 'third_party/pocketfft' 2023-03-08T02:46:16.3682326Z Entering 'third_party/protobuf' 2023-03-08T02:46:16.3724741Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:16.3762842Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:16.3802859Z Entering 'third_party/psimd' 2023-03-08T02:46:16.3842205Z Entering 'third_party/pthreadpool' 2023-03-08T02:46:16.3880575Z Entering 'third_party/pybind11' 2023-03-08T02:46:16.3919934Z Entering 'third_party/python-enum' 2023-03-08T02:46:16.3958338Z Entering 'third_party/python-peachpy' 2023-03-08T02:46:16.3996462Z Entering 'third_party/python-six' 2023-03-08T02:46:16.4034631Z Entering 'third_party/sleef' 2023-03-08T02:46:16.4073421Z Entering 'third_party/tbb' 2023-03-08T02:46:16.4113902Z Entering 'third_party/tensorpipe' 2023-03-08T02:46:16.4152072Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:16.4190393Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:16.4227959Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:16.4265546Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:16.4302228Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:16.4344430Z Entering 'third_party/zstd' 2023-03-08T02:46:16.4394922Z ##[endgroup] 2023-03-08T02:46:16.4395502Z ##[group]Persisting credentials for submodules 2023-03-08T02:46:16.4400173Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2023-03-08T02:46:16.4665470Z Entering 'android/libs/fbjni' 2023-03-08T02:46:16.4703432Z Entering 'third_party/FP16' 2023-03-08T02:46:16.4740206Z Entering 'third_party/FXdiv' 2023-03-08T02:46:16.4776977Z Entering 'third_party/NNPACK' 2023-03-08T02:46:16.4814207Z Entering 'third_party/QNNPACK' 2023-03-08T02:46:16.4851150Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T02:46:16.4888353Z Entering 'third_party/XNNPACK' 2023-03-08T02:46:16.4938693Z Entering 'third_party/benchmark' 2023-03-08T02:46:16.4975641Z Entering 'third_party/cpuinfo' 2023-03-08T02:46:16.5013275Z Entering 'third_party/cub' 2023-03-08T02:46:16.5050022Z Entering 'third_party/cudnn_frontend' 2023-03-08T02:46:16.5093546Z Entering 'third_party/cutlass' 2023-03-08T02:46:16.5136813Z Entering 'third_party/eigen' 2023-03-08T02:46:16.5175522Z Entering 'third_party/fbgemm' 2023-03-08T02:46:16.5212791Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:46:16.5249019Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:46:16.5284886Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:46:16.5328275Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:46:16.5363936Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:46:16.5401883Z Entering 'third_party/flatbuffers' 2023-03-08T02:46:16.5441104Z Entering 'third_party/fmt' 2023-03-08T02:46:16.5478691Z Entering 'third_party/foxi' 2023-03-08T02:46:16.5515670Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:46:16.5552212Z Entering 'third_party/gloo' 2023-03-08T02:46:16.5588957Z Entering 'third_party/googletest' 2023-03-08T02:46:16.5625279Z Entering 'third_party/ideep' 2023-03-08T02:46:16.5660962Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T02:46:16.5705202Z Entering 'third_party/ios-cmake' 2023-03-08T02:46:16.5741874Z Entering 'third_party/ittapi' 2023-03-08T02:46:16.5777895Z Entering 'third_party/kineto' 2023-03-08T02:46:16.5814357Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:46:16.5850685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:46:16.5889345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:46:16.5925756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:46:16.5962583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:46:16.5998139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:46:16.6035716Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:46:16.6072059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:46:16.6108130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:46:16.6145665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:46:16.6183864Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:46:16.6220156Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:46:16.6258237Z Entering 'third_party/nccl/nccl' 2023-03-08T02:46:16.6293963Z Entering 'third_party/neon2sse' 2023-03-08T02:46:16.6330458Z Entering 'third_party/nlohmann' 2023-03-08T02:46:16.6367873Z Entering 'third_party/onnx' 2023-03-08T02:46:16.6418542Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:16.6455947Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:16.6494797Z Entering 'third_party/onnx-tensorrt' 2023-03-08T02:46:16.6531377Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:16.6572109Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:16.6609296Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:16.6644699Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:16.6686577Z Entering 'third_party/pocketfft' 2023-03-08T02:46:16.6722709Z Entering 'third_party/protobuf' 2023-03-08T02:46:16.6762311Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:16.6798406Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:16.6836502Z Entering 'third_party/psimd' 2023-03-08T02:46:16.6872919Z Entering 'third_party/pthreadpool' 2023-03-08T02:46:16.6909072Z Entering 'third_party/pybind11' 2023-03-08T02:46:16.6945496Z Entering 'third_party/python-enum' 2023-03-08T02:46:16.6982622Z Entering 'third_party/python-peachpy' 2023-03-08T02:46:16.7018657Z Entering 'third_party/python-six' 2023-03-08T02:46:16.7056275Z Entering 'third_party/sleef' 2023-03-08T02:46:16.7092093Z Entering 'third_party/tbb' 2023-03-08T02:46:16.7131624Z Entering 'third_party/tensorpipe' 2023-03-08T02:46:16.7167328Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:16.7204188Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:16.7239853Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:16.7276860Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:16.7311883Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:16.7350940Z Entering 'third_party/zstd' 2023-03-08T02:46:16.7400639Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2023-03-08T02:46:16.7662039Z Entering 'android/libs/fbjni' 2023-03-08T02:46:16.7696476Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2023-03-08T02:46:16.7714139Z Entering 'third_party/FP16' 2023-03-08T02:46:16.7749522Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2023-03-08T02:46:16.7767215Z Entering 'third_party/FXdiv' 2023-03-08T02:46:16.7802522Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2023-03-08T02:46:16.7819093Z Entering 'third_party/NNPACK' 2023-03-08T02:46:16.7854073Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2023-03-08T02:46:16.7871853Z Entering 'third_party/QNNPACK' 2023-03-08T02:46:16.7906133Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2023-03-08T02:46:16.7923576Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T02:46:16.7958304Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2023-03-08T02:46:16.7975519Z Entering 'third_party/XNNPACK' 2023-03-08T02:46:16.8010409Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2023-03-08T02:46:16.8042225Z Entering 'third_party/benchmark' 2023-03-08T02:46:16.8076450Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2023-03-08T02:46:16.8094509Z Entering 'third_party/cpuinfo' 2023-03-08T02:46:16.8129228Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2023-03-08T02:46:16.8147640Z Entering 'third_party/cub' 2023-03-08T02:46:16.8182231Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2023-03-08T02:46:16.8200169Z Entering 'third_party/cudnn_frontend' 2023-03-08T02:46:16.8234139Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2023-03-08T02:46:16.8257783Z Entering 'third_party/cutlass' 2023-03-08T02:46:16.8293038Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2023-03-08T02:46:16.8317856Z Entering 'third_party/eigen' 2023-03-08T02:46:16.8352147Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2023-03-08T02:46:16.8372160Z Entering 'third_party/fbgemm' 2023-03-08T02:46:16.8406981Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2023-03-08T02:46:16.8423778Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:46:16.8458237Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2023-03-08T02:46:16.8476542Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:46:16.8510517Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2023-03-08T02:46:16.8528018Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:46:16.8561430Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2023-03-08T02:46:16.8586767Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:46:16.8619466Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2023-03-08T02:46:16.8636979Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:46:16.8669770Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2023-03-08T02:46:16.8688292Z Entering 'third_party/flatbuffers' 2023-03-08T02:46:16.8722702Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2023-03-08T02:46:16.8741434Z Entering 'third_party/fmt' 2023-03-08T02:46:16.8775514Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2023-03-08T02:46:16.8792798Z Entering 'third_party/foxi' 2023-03-08T02:46:16.8826637Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2023-03-08T02:46:16.8844119Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:46:16.8878347Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2023-03-08T02:46:16.8895722Z Entering 'third_party/gloo' 2023-03-08T02:46:16.8929385Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2023-03-08T02:46:16.8946906Z Entering 'third_party/googletest' 2023-03-08T02:46:16.8980469Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2023-03-08T02:46:16.8998701Z Entering 'third_party/ideep' 2023-03-08T02:46:16.9032899Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2023-03-08T02:46:16.9049204Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T02:46:16.9082012Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2023-03-08T02:46:16.9108913Z Entering 'third_party/ios-cmake' 2023-03-08T02:46:16.9142107Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2023-03-08T02:46:16.9160619Z Entering 'third_party/ittapi' 2023-03-08T02:46:16.9193879Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2023-03-08T02:46:16.9211583Z Entering 'third_party/kineto' 2023-03-08T02:46:16.9244912Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2023-03-08T02:46:16.9262891Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:46:16.9297792Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2023-03-08T02:46:16.9315397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:46:16.9350504Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2023-03-08T02:46:16.9369580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:46:16.9404440Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2023-03-08T02:46:16.9421898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:46:16.9456622Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2023-03-08T02:46:16.9474632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:46:16.9509254Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2023-03-08T02:46:16.9525257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:46:16.9561416Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2023-03-08T02:46:16.9580026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:46:16.9614296Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2023-03-08T02:46:16.9632733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:46:16.9665796Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2023-03-08T02:46:16.9683788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:46:16.9717518Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2023-03-08T02:46:16.9735937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:46:16.9770339Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2023-03-08T02:46:16.9789892Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:46:16.9823086Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2023-03-08T02:46:16.9840705Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:46:16.9874313Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2023-03-08T02:46:16.9893380Z Entering 'third_party/nccl/nccl' 2023-03-08T02:46:16.9927269Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2023-03-08T02:46:16.9945091Z Entering 'third_party/neon2sse' 2023-03-08T02:46:16.9978922Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2023-03-08T02:46:16.9996662Z Entering 'third_party/nlohmann' 2023-03-08T02:46:17.0030467Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2023-03-08T02:46:17.0049146Z Entering 'third_party/onnx' 2023-03-08T02:46:17.0083345Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2023-03-08T02:46:17.0116092Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.0150510Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-03-08T02:46:17.0167661Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.0200627Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-03-08T02:46:17.0220473Z Entering 'third_party/onnx-tensorrt' 2023-03-08T02:46:17.0254064Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2023-03-08T02:46:17.0270870Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:17.0304279Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2023-03-08T02:46:17.0326814Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.0360523Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-03-08T02:46:17.0377524Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.0410880Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-03-08T02:46:17.0427500Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.0463433Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-03-08T02:46:17.0485625Z Entering 'third_party/pocketfft' 2023-03-08T02:46:17.0519157Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2023-03-08T02:46:17.0536530Z Entering 'third_party/protobuf' 2023-03-08T02:46:17.0570329Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2023-03-08T02:46:17.0591650Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:17.0624502Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2023-03-08T02:46:17.0641928Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:17.0675469Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2023-03-08T02:46:17.0694677Z Entering 'third_party/psimd' 2023-03-08T02:46:17.0727792Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2023-03-08T02:46:17.0745192Z Entering 'third_party/pthreadpool' 2023-03-08T02:46:17.0778586Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2023-03-08T02:46:17.0797081Z Entering 'third_party/pybind11' 2023-03-08T02:46:17.0829916Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2023-03-08T02:46:17.0848277Z Entering 'third_party/python-enum' 2023-03-08T02:46:17.0881794Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2023-03-08T02:46:17.0899714Z Entering 'third_party/python-peachpy' 2023-03-08T02:46:17.0933075Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2023-03-08T02:46:17.0951637Z Entering 'third_party/python-six' 2023-03-08T02:46:17.0984491Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2023-03-08T02:46:17.1003076Z Entering 'third_party/sleef' 2023-03-08T02:46:17.1036206Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2023-03-08T02:46:17.1054910Z Entering 'third_party/tbb' 2023-03-08T02:46:17.1088726Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2023-03-08T02:46:17.1108656Z Entering 'third_party/tensorpipe' 2023-03-08T02:46:17.1142104Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2023-03-08T02:46:17.1159126Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:17.1193335Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2023-03-08T02:46:17.1210142Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:17.1243018Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2023-03-08T02:46:17.1260972Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:17.1293981Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2023-03-08T02:46:17.1311515Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:17.1344773Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2023-03-08T02:46:17.1361651Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.1396020Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-03-08T02:46:17.1418663Z Entering 'third_party/zstd' 2023-03-08T02:46:17.1451751Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2023-03-08T02:46:17.1835820Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2023-03-08T02:46:17.2106565Z Entering 'android/libs/fbjni' 2023-03-08T02:46:17.2144928Z Entering 'third_party/FP16' 2023-03-08T02:46:17.2183637Z Entering 'third_party/FXdiv' 2023-03-08T02:46:17.2221306Z Entering 'third_party/NNPACK' 2023-03-08T02:46:17.2259328Z Entering 'third_party/QNNPACK' 2023-03-08T02:46:17.2297492Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T02:46:17.2336227Z Entering 'third_party/XNNPACK' 2023-03-08T02:46:17.2386444Z Entering 'third_party/benchmark' 2023-03-08T02:46:17.2424288Z Entering 'third_party/cpuinfo' 2023-03-08T02:46:17.2462700Z Entering 'third_party/cub' 2023-03-08T02:46:17.2500401Z Entering 'third_party/cudnn_frontend' 2023-03-08T02:46:17.2543472Z Entering 'third_party/cutlass' 2023-03-08T02:46:17.2587202Z Entering 'third_party/eigen' 2023-03-08T02:46:17.2626707Z Entering 'third_party/fbgemm' 2023-03-08T02:46:17.2664379Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:46:17.2703005Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:46:17.2740097Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:46:17.2785266Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:46:17.2822210Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:46:17.2860358Z Entering 'third_party/flatbuffers' 2023-03-08T02:46:17.2900603Z Entering 'third_party/fmt' 2023-03-08T02:46:17.2938033Z Entering 'third_party/foxi' 2023-03-08T02:46:17.2976148Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:46:17.3013619Z Entering 'third_party/gloo' 2023-03-08T02:46:17.3051598Z Entering 'third_party/googletest' 2023-03-08T02:46:17.3089585Z Entering 'third_party/ideep' 2023-03-08T02:46:17.3126462Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T02:46:17.3173309Z Entering 'third_party/ios-cmake' 2023-03-08T02:46:17.3214332Z Entering 'third_party/ittapi' 2023-03-08T02:46:17.3253387Z Entering 'third_party/kineto' 2023-03-08T02:46:17.3291319Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:46:17.3329903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:46:17.3369936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:46:17.3407286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:46:17.3445062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:46:17.3482521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:46:17.3522822Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:46:17.3561842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:46:17.3599600Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:46:17.3638267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:46:17.3677368Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:46:17.3715179Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:46:17.3755319Z Entering 'third_party/nccl/nccl' 2023-03-08T02:46:17.3794396Z Entering 'third_party/neon2sse' 2023-03-08T02:46:17.3831760Z Entering 'third_party/nlohmann' 2023-03-08T02:46:17.3871872Z Entering 'third_party/onnx' 2023-03-08T02:46:17.3924439Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.3962257Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.4002751Z Entering 'third_party/onnx-tensorrt' 2023-03-08T02:46:17.4040351Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:17.4082268Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.4121493Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.4160277Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.4205833Z Entering 'third_party/pocketfft' 2023-03-08T02:46:17.4244568Z Entering 'third_party/protobuf' 2023-03-08T02:46:17.4288519Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:17.4327794Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:17.4368415Z Entering 'third_party/psimd' 2023-03-08T02:46:17.4407124Z Entering 'third_party/pthreadpool' 2023-03-08T02:46:17.4445402Z Entering 'third_party/pybind11' 2023-03-08T02:46:17.4483650Z Entering 'third_party/python-enum' 2023-03-08T02:46:17.4523280Z Entering 'third_party/python-peachpy' 2023-03-08T02:46:17.4564945Z Entering 'third_party/python-six' 2023-03-08T02:46:17.4603817Z Entering 'third_party/sleef' 2023-03-08T02:46:17.4643519Z Entering 'third_party/tbb' 2023-03-08T02:46:17.4685199Z Entering 'third_party/tensorpipe' 2023-03-08T02:46:17.4724135Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:17.4763183Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:17.4801375Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:17.4840568Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:17.4878090Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.4920805Z Entering 'third_party/zstd' 2023-03-08T02:46:17.4972281Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2023-03-08T02:46:17.5243014Z Entering 'android/libs/fbjni' 2023-03-08T02:46:17.5284641Z Entering 'third_party/FP16' 2023-03-08T02:46:17.5323828Z Entering 'third_party/FXdiv' 2023-03-08T02:46:17.5363398Z Entering 'third_party/NNPACK' 2023-03-08T02:46:17.5402710Z Entering 'third_party/QNNPACK' 2023-03-08T02:46:17.5441913Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T02:46:17.5481202Z Entering 'third_party/XNNPACK' 2023-03-08T02:46:17.5533593Z Entering 'third_party/benchmark' 2023-03-08T02:46:17.5573574Z Entering 'third_party/cpuinfo' 2023-03-08T02:46:17.5613768Z Entering 'third_party/cub' 2023-03-08T02:46:17.5653267Z Entering 'third_party/cudnn_frontend' 2023-03-08T02:46:17.5699598Z Entering 'third_party/cutlass' 2023-03-08T02:46:17.5744765Z Entering 'third_party/eigen' 2023-03-08T02:46:17.5785205Z Entering 'third_party/fbgemm' 2023-03-08T02:46:17.5824705Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T02:46:17.5862935Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T02:46:17.5900211Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T02:46:17.5946524Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T02:46:17.5983378Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T02:46:17.6024081Z Entering 'third_party/flatbuffers' 2023-03-08T02:46:17.6063921Z Entering 'third_party/fmt' 2023-03-08T02:46:17.6103026Z Entering 'third_party/foxi' 2023-03-08T02:46:17.6141421Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T02:46:17.6180695Z Entering 'third_party/gloo' 2023-03-08T02:46:17.6220059Z Entering 'third_party/googletest' 2023-03-08T02:46:17.6260189Z Entering 'third_party/ideep' 2023-03-08T02:46:17.6297946Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T02:46:17.6346038Z Entering 'third_party/ios-cmake' 2023-03-08T02:46:17.6384588Z Entering 'third_party/ittapi' 2023-03-08T02:46:17.6422809Z Entering 'third_party/kineto' 2023-03-08T02:46:17.6461528Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T02:46:17.6499954Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T02:46:17.6540178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T02:46:17.6578475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T02:46:17.6617196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T02:46:17.6654697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T02:46:17.6694841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T02:46:17.6733553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T02:46:17.6773491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T02:46:17.6813355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T02:46:17.6853565Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T02:46:17.6892168Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T02:46:17.6932476Z Entering 'third_party/nccl/nccl' 2023-03-08T02:46:17.6970799Z Entering 'third_party/neon2sse' 2023-03-08T02:46:17.7008505Z Entering 'third_party/nlohmann' 2023-03-08T02:46:17.7049279Z Entering 'third_party/onnx' 2023-03-08T02:46:17.7101701Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.7141128Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.7182463Z Entering 'third_party/onnx-tensorrt' 2023-03-08T02:46:17.7221140Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T02:46:17.7262517Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T02:46:17.7300426Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T02:46:17.7336962Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.7381276Z Entering 'third_party/pocketfft' 2023-03-08T02:46:17.7419345Z Entering 'third_party/protobuf' 2023-03-08T02:46:17.7461889Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T02:46:17.7500493Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T02:46:17.7540664Z Entering 'third_party/psimd' 2023-03-08T02:46:17.7579675Z Entering 'third_party/pthreadpool' 2023-03-08T02:46:17.7617608Z Entering 'third_party/pybind11' 2023-03-08T02:46:17.7656666Z Entering 'third_party/python-enum' 2023-03-08T02:46:17.7694998Z Entering 'third_party/python-peachpy' 2023-03-08T02:46:17.7733546Z Entering 'third_party/python-six' 2023-03-08T02:46:17.7771825Z Entering 'third_party/sleef' 2023-03-08T02:46:17.7811359Z Entering 'third_party/tbb' 2023-03-08T02:46:17.7851668Z Entering 'third_party/tensorpipe' 2023-03-08T02:46:17.7889801Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T02:46:17.7928231Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T02:46:17.7965230Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T02:46:17.8003855Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T02:46:17.8041855Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T02:46:17.8082779Z Entering 'third_party/zstd' 2023-03-08T02:46:17.8131207Z ##[endgroup] 2023-03-08T02:46:17.8178703Z [command]/usr/bin/git log -1 --format='%H' 2023-03-08T02:46:17.8212858Z 'c88aa336aa0734f42b4d9db7f624d6cfd9b5065e' 2023-03-08T02:46:17.8401065Z Prepare all required actions 2023-03-08T02:46:17.8401933Z Getting action download info 2023-03-08T02:46:18.0856749Z ##[group]Run ./.github/actions/setup-linux 2023-03-08T02:46:18.0856996Z env: 2023-03-08T02:46:18.0857222Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:18.0857455Z ##[endgroup] 2023-03-08T02:46:18.0876021Z ##[group]Run set -euo pipefail 2023-03-08T02:46:18.0876317Z set -euo pipefail 2023-03-08T02:46:18.0876582Z function get_ec2_metadata() { 2023-03-08T02:46:18.0876890Z  # Pulled from instance metadata endpoint for EC2 2023-03-08T02:46:18.0877334Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2023-03-08T02:46:18.0877691Z  category=$1 2023-03-08T02:46:18.0877994Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2023-03-08T02:46:18.0878276Z } 2023-03-08T02:46:18.0878523Z echo "ami-id: $(get_ec2_metadata ami-id)" 2023-03-08T02:46:18.0878886Z echo "instance-id: $(get_ec2_metadata instance-id)" 2023-03-08T02:46:18.0879232Z echo "instance-type: $(get_ec2_metadata instance-type)" 2023-03-08T02:46:18.0879524Z echo "system info $(uname -a)" 2023-03-08T02:46:18.0897566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:46:18.0897846Z env: 2023-03-08T02:46:18.0898052Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:18.0898287Z ##[endgroup] 2023-03-08T02:46:18.1041124Z curl: (22) The requested URL returned error: 404 Not Found 2023-03-08T02:46:18.1051574Z ami-id: 2023-03-08T02:46:18.1139226Z curl: (22) The requested URL returned error: 404 Not Found 2023-03-08T02:46:18.1148108Z instance-id: 2023-03-08T02:46:18.1232053Z curl: (22) The requested URL returned error: 404 Not Found 2023-03-08T02:46:18.1240772Z instance-type: 2023-03-08T02:46:18.1251972Z system info Linux gh-ci-gcp-a100-11 5.15.0-1030-gcp #37~20.04.1-Ubuntu SMP Mon Feb 20 04:30:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux 2023-03-08T02:46:18.1271486Z ##[group]Run if systemctl is-active --quiet docker; then 2023-03-08T02:46:18.1271850Z if systemctl is-active --quiet docker; then 2023-03-08T02:46:18.1272152Z  echo "Docker daemon is running..."; 2023-03-08T02:46:18.1272398Z else 2023-03-08T02:46:18.1272672Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2023-03-08T02:46:18.1272949Z fi 2023-03-08T02:46:18.1289826Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:46:18.1290079Z env: 2023-03-08T02:46:18.1290295Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:18.1290525Z ##[endgroup] 2023-03-08T02:46:18.1355864Z Docker daemon is running... 2023-03-08T02:46:18.1389669Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-03-08T02:46:18.1389947Z with: 2023-03-08T02:46:18.1390145Z shell: bash 2023-03-08T02:46:18.1390359Z timeout_minutes: 5 2023-03-08T02:46:18.1390592Z max_attempts: 3 2023-03-08T02:46:18.1390810Z retry_wait_seconds: 30 2023-03-08T02:46:18.1391504Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2023-03-08T02:46:18.1392053Z polling_interval_seconds: 1 2023-03-08T02:46:18.1392299Z warning_on_retry: true 2023-03-08T02:46:18.1392520Z continue_on_error: false 2023-03-08T02:46:18.1392734Z env: 2023-03-08T02:46:18.1392977Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:18.1393205Z AWS_RETRY_MODE: standard 2023-03-08T02:46:18.1393436Z AWS_MAX_ATTEMPTS: 5 2023-03-08T02:46:18.1393676Z AWS_DEFAULT_REGION: us-east-1 2023-03-08T02:46:18.1393908Z ##[endgroup] 2023-03-08T02:46:20.4493108Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json. 2023-03-08T02:46:20.4493623Z Configure a credential helper to remove this warning. See 2023-03-08T02:46:20.4494748Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2023-03-08T02:46:20.4495005Z 2023-03-08T02:46:20.4496301Z Login Succeeded 2023-03-08T02:46:21.2026972Z Command completed after 1 attempt(s). 2023-03-08T02:46:21.2092738Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-03-08T02:46:21.2093132Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-03-08T02:46:21.2093462Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-03-08T02:46:21.2111557Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:46:21.2111822Z env: 2023-03-08T02:46:21.2112042Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:21.2112273Z ##[endgroup] 2023-03-08T02:46:21.2214663Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2023-03-08T02:46:21.2214973Z with: 2023-03-08T02:46:21.2215421Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:46:21.2215873Z env: 2023-03-08T02:46:21.2216085Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:21.2216317Z ##[endgroup] 2023-03-08T02:46:21.2230359Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-03-08T02:46:21.2230695Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-03-08T02:46:21.2231027Z # ignore output since only exit code is used for conditional 2023-03-08T02:46:21.2231381Z # only pull docker image if it's not available locally 2023-03-08T02:46:21.2231737Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2023-03-08T02:46:21.2232112Z  retry docker pull "${DOCKER_IMAGE}" 2023-03-08T02:46:21.2232355Z fi 2023-03-08T02:46:21.2248470Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:46:21.2248747Z env: 2023-03-08T02:46:21.2248970Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:46:21.2249445Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:46:21.2249897Z ##[endgroup] 2023-03-08T02:46:21.8196811Z fadf10ce2391ba6c62d3bef60b68f92b635268f7: Pulling from pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7 2023-03-08T02:46:21.8197654Z 456d651ccb27: Pulling fs layer 2023-03-08T02:46:21.8198129Z 2ea9ef18f7bc: Pulling fs layer 2023-03-08T02:46:21.8198551Z 94a24fc23c7c: Pulling fs layer 2023-03-08T02:46:21.8199032Z 2c4f28751241: Pulling fs layer 2023-03-08T02:46:21.8199442Z a9e5e49da50d: Pulling fs layer 2023-03-08T02:46:21.8199888Z 1aece8b4bd63: Pulling fs layer 2023-03-08T02:46:21.8200275Z 8030ee497010: Pulling fs layer 2023-03-08T02:46:21.8200599Z a41332b1bef3: Pulling fs layer 2023-03-08T02:46:21.8200915Z aed8824e0575: Pulling fs layer 2023-03-08T02:46:21.8201249Z ecf188d26601: Pulling fs layer 2023-03-08T02:46:21.8201629Z 8c74646e5134: Pulling fs layer 2023-03-08T02:46:21.8202266Z 2c4f28751241: Waiting 2023-03-08T02:46:21.8202627Z cb9c93f25a34: Pulling fs layer 2023-03-08T02:46:21.8203005Z a9e5e49da50d: Waiting 2023-03-08T02:46:21.8203423Z 8e9933b60756: Pulling fs layer 2023-03-08T02:46:21.8203806Z 74d24d3330f0: Pulling fs layer 2023-03-08T02:46:21.8204160Z 1aece8b4bd63: Waiting 2023-03-08T02:46:21.8204517Z a90c1a192b83: Pulling fs layer 2023-03-08T02:46:21.8204881Z a41332b1bef3: Waiting 2023-03-08T02:46:21.8205221Z 8fb22cf02bcd: Pulling fs layer 2023-03-08T02:46:21.8205547Z 8030ee497010: Waiting 2023-03-08T02:46:21.8205886Z 92854560e3ba: Pulling fs layer 2023-03-08T02:46:21.8206246Z 444c90fcb6b3: Pulling fs layer 2023-03-08T02:46:21.8206664Z 849d613e1f4f: Pulling fs layer 2023-03-08T02:46:21.8207055Z 09249daa7056: Pulling fs layer 2023-03-08T02:46:21.8207448Z cb9c93f25a34: Waiting 2023-03-08T02:46:21.8207831Z 75558b4a699c: Pulling fs layer 2023-03-08T02:46:21.8208195Z bbe1302c0658: Pulling fs layer 2023-03-08T02:46:21.8208902Z 8c74646e5134: Waiting 2023-03-08T02:46:21.8209267Z 3ddb6b2ea880: Pulling fs layer 2023-03-08T02:46:21.8209622Z ecf188d26601: Waiting 2023-03-08T02:46:21.8209996Z e0726beb3694: Pulling fs layer 2023-03-08T02:46:21.8210373Z 92854560e3ba: Waiting 2023-03-08T02:46:21.8210750Z b5a9a355bdf8: Pulling fs layer 2023-03-08T02:46:21.8211138Z a90c1a192b83: Waiting 2023-03-08T02:46:21.8211522Z 8e9933b60756: Waiting 2023-03-08T02:46:21.8211889Z b5b2af6db879: Pulling fs layer 2023-03-08T02:46:21.8212284Z 444c90fcb6b3: Waiting 2023-03-08T02:46:21.8212655Z 849d613e1f4f: Waiting 2023-03-08T02:46:21.8212995Z 8fb22cf02bcd: Waiting 2023-03-08T02:46:21.8213442Z bbe1302c0658: Waiting 2023-03-08T02:46:21.8213770Z 75558b4a699c: Waiting 2023-03-08T02:46:21.8214080Z 3ddb6b2ea880: Waiting 2023-03-08T02:46:21.8214713Z 09249daa7056: Waiting 2023-03-08T02:46:21.8215076Z aed8824e0575: Waiting 2023-03-08T02:46:21.8215421Z b5a9a355bdf8: Waiting 2023-03-08T02:46:21.8215774Z 74d24d3330f0: Waiting 2023-03-08T02:46:21.8216132Z ebb48cc5f2f8: Pulling fs layer 2023-03-08T02:46:21.8216504Z b5b2af6db879: Waiting 2023-03-08T02:46:21.8216872Z 4640a3951080: Pulling fs layer 2023-03-08T02:46:21.8217303Z 42168f59750f: Pulling fs layer 2023-03-08T02:46:21.8217708Z d05e3de10596: Pulling fs layer 2023-03-08T02:46:21.8218112Z 5849c6bb9570: Pulling fs layer 2023-03-08T02:46:21.8218471Z 4640a3951080: Waiting 2023-03-08T02:46:21.8218854Z 8dfe045dbd77: Pulling fs layer 2023-03-08T02:46:21.8219253Z 86320342bcd0: Pulling fs layer 2023-03-08T02:46:21.8219629Z d05e3de10596: Waiting 2023-03-08T02:46:21.8220024Z f40bd470d742: Pulling fs layer 2023-03-08T02:46:21.8220406Z ebb48cc5f2f8: Waiting 2023-03-08T02:46:21.8220794Z 67b32c0fb809: Pulling fs layer 2023-03-08T02:46:21.8221179Z 585dfa7633d3: Pulling fs layer 2023-03-08T02:46:21.8221521Z f40bd470d742: Waiting 2023-03-08T02:46:21.8221861Z 8dfe045dbd77: Waiting 2023-03-08T02:46:21.8222231Z 5392ad3482eb: Pulling fs layer 2023-03-08T02:46:21.8222592Z 86320342bcd0: Waiting 2023-03-08T02:46:21.8222967Z 8c656d07c114: Pulling fs layer 2023-03-08T02:46:21.8223433Z 67b32c0fb809: Waiting 2023-03-08T02:46:21.8223748Z 585dfa7633d3: Waiting 2023-03-08T02:46:21.8224112Z f4b9815ad375: Pulling fs layer 2023-03-08T02:46:21.8224492Z 559c0a582918: Pulling fs layer 2023-03-08T02:46:21.8224848Z 5392ad3482eb: Waiting 2023-03-08T02:46:21.8225223Z ea3a51f74b8e: Pulling fs layer 2023-03-08T02:46:21.8225632Z a5576526a324: Pulling fs layer 2023-03-08T02:46:21.8226055Z 559c0a582918: Waiting 2023-03-08T02:46:21.8226409Z ba972896ba4a: Pulling fs layer 2023-03-08T02:46:21.8226781Z 8c656d07c114: Waiting 2023-03-08T02:46:21.8227323Z ea3a51f74b8e: Waiting 2023-03-08T02:46:21.8227667Z a5576526a324: Waiting 2023-03-08T02:46:21.8228045Z 3c800a976df4: Pulling fs layer 2023-03-08T02:46:21.8228445Z 5537b4d41660: Pulling fs layer 2023-03-08T02:46:21.8228831Z ba972896ba4a: Waiting 2023-03-08T02:46:21.8229212Z 810cfd043ef3: Pulling fs layer 2023-03-08T02:46:21.8229633Z 7145aecfb4a6: Pulling fs layer 2023-03-08T02:46:21.8230030Z 34a279bd458a: Pulling fs layer 2023-03-08T02:46:21.8230416Z 2454956ec55e: Pulling fs layer 2023-03-08T02:46:21.8230816Z 3b5185ebd5e7: Pulling fs layer 2023-03-08T02:46:21.8231185Z cdcd60c75136: Pulling fs layer 2023-03-08T02:46:21.8231575Z 898f0f4942b4: Pulling fs layer 2023-03-08T02:46:21.8231955Z 3c800a976df4: Waiting 2023-03-08T02:46:21.8232328Z 784bb0b58a0d: Pulling fs layer 2023-03-08T02:46:21.8232694Z 3b5185ebd5e7: Waiting 2023-03-08T02:46:21.8233056Z 2454956ec55e: Waiting 2023-03-08T02:46:21.8233465Z cdcd60c75136: Waiting 2023-03-08T02:46:21.8233834Z 7145aecfb4a6: Waiting 2023-03-08T02:46:21.8234251Z a7d3b22585a1: Pulling fs layer 2023-03-08T02:46:21.8234636Z 5537b4d41660: Waiting 2023-03-08T02:46:21.8235001Z 898f0f4942b4: Waiting 2023-03-08T02:46:21.8235392Z b916fdfdfcb5: Pulling fs layer 2023-03-08T02:46:21.8235795Z 810cfd043ef3: Waiting 2023-03-08T02:46:21.8236196Z b85022ab5920: Pulling fs layer 2023-03-08T02:46:21.8236606Z a7d3b22585a1: Waiting 2023-03-08T02:46:21.8237005Z 5b2adf07ae20: Pulling fs layer 2023-03-08T02:46:21.8237423Z 7818bcc94020: Pulling fs layer 2023-03-08T02:46:21.8238085Z b85022ab5920: Waiting 2023-03-08T02:46:21.8238444Z 5b2adf07ae20: Waiting 2023-03-08T02:46:21.8238797Z 7818bcc94020: Waiting 2023-03-08T02:46:22.3705126Z 2ea9ef18f7bc: Download complete 2023-03-08T02:46:22.6628656Z 2c4f28751241: Verifying Checksum 2023-03-08T02:46:22.6628968Z 2c4f28751241: Download complete 2023-03-08T02:46:22.6708759Z 456d651ccb27: Verifying Checksum 2023-03-08T02:46:22.6709050Z 456d651ccb27: Download complete 2023-03-08T02:46:22.9376638Z 94a24fc23c7c: Verifying Checksum 2023-03-08T02:46:22.9377095Z 94a24fc23c7c: Download complete 2023-03-08T02:46:22.9556176Z a9e5e49da50d: Download complete 2023-03-08T02:46:23.2503604Z a41332b1bef3: Download complete 2023-03-08T02:46:23.2894613Z 8030ee497010: Download complete 2023-03-08T02:46:23.3599497Z 456d651ccb27: Pull complete 2023-03-08T02:46:23.5413583Z aed8824e0575: Download complete 2023-03-08T02:46:23.5765905Z 2ea9ef18f7bc: Pull complete 2023-03-08T02:46:23.8886885Z 8c74646e5134: Verifying Checksum 2023-03-08T02:46:23.8887425Z 8c74646e5134: Download complete 2023-03-08T02:46:24.2081435Z cb9c93f25a34: Verifying Checksum 2023-03-08T02:46:24.2082133Z cb9c93f25a34: Download complete 2023-03-08T02:46:24.6161624Z 94a24fc23c7c: Pull complete 2023-03-08T02:46:24.6781411Z 2c4f28751241: Pull complete 2023-03-08T02:46:24.7413722Z a9e5e49da50d: Pull complete 2023-03-08T02:46:26.9773235Z 8e9933b60756: Verifying Checksum 2023-03-08T02:46:26.9773691Z 8e9933b60756: Download complete 2023-03-08T02:46:27.3736777Z 74d24d3330f0: Download complete 2023-03-08T02:46:27.6983275Z a90c1a192b83: Verifying Checksum 2023-03-08T02:46:27.6983629Z a90c1a192b83: Download complete 2023-03-08T02:46:28.0167315Z 8fb22cf02bcd: Verifying Checksum 2023-03-08T02:46:28.0167760Z 8fb22cf02bcd: Download complete 2023-03-08T02:46:29.3357874Z 92854560e3ba: Verifying Checksum 2023-03-08T02:46:29.3358250Z 92854560e3ba: Download complete 2023-03-08T02:46:29.6347082Z 444c90fcb6b3: Verifying Checksum 2023-03-08T02:46:29.6347741Z 444c90fcb6b3: Download complete 2023-03-08T02:46:29.9277444Z 849d613e1f4f: Verifying Checksum 2023-03-08T02:46:29.9277928Z 849d613e1f4f: Download complete 2023-03-08T02:46:30.2254121Z 09249daa7056: Verifying Checksum 2023-03-08T02:46:30.2254723Z 09249daa7056: Download complete 2023-03-08T02:46:36.7528106Z 1aece8b4bd63: Verifying Checksum 2023-03-08T02:46:36.7528463Z 1aece8b4bd63: Download complete 2023-03-08T02:46:37.0801266Z bbe1302c0658: Verifying Checksum 2023-03-08T02:46:37.0801583Z bbe1302c0658: Download complete 2023-03-08T02:46:37.3702323Z 3ddb6b2ea880: Download complete 2023-03-08T02:46:37.6794776Z e0726beb3694: Download complete 2023-03-08T02:46:37.9714071Z b5a9a355bdf8: Verifying Checksum 2023-03-08T02:46:37.9714583Z b5a9a355bdf8: Download complete 2023-03-08T02:46:38.2681527Z b5b2af6db879: Download complete 2023-03-08T02:46:39.7311633Z ebb48cc5f2f8: Verifying Checksum 2023-03-08T02:46:39.7312154Z ebb48cc5f2f8: Download complete 2023-03-08T02:46:40.0108774Z 4640a3951080: Verifying Checksum 2023-03-08T02:46:40.0109333Z 4640a3951080: Download complete 2023-03-08T02:46:40.2959642Z 42168f59750f: Verifying Checksum 2023-03-08T02:46:40.2960107Z 42168f59750f: Download complete 2023-03-08T02:46:40.7436916Z d05e3de10596: Verifying Checksum 2023-03-08T02:46:40.7437293Z d05e3de10596: Download complete 2023-03-08T02:46:41.0456231Z 5849c6bb9570: Verifying Checksum 2023-03-08T02:46:41.0456591Z 5849c6bb9570: Download complete 2023-03-08T02:46:41.3539335Z 8dfe045dbd77: Download complete 2023-03-08T02:46:41.4829664Z ecf188d26601: Verifying Checksum 2023-03-08T02:46:41.4829988Z ecf188d26601: Download complete 2023-03-08T02:46:41.7625238Z f40bd470d742: Verifying Checksum 2023-03-08T02:46:41.7625763Z f40bd470d742: Download complete 2023-03-08T02:46:42.0691279Z 67b32c0fb809: Download complete 2023-03-08T02:46:42.9681664Z 585dfa7633d3: Verifying Checksum 2023-03-08T02:46:42.9682078Z 585dfa7633d3: Download complete 2023-03-08T02:46:43.2675984Z 5392ad3482eb: Download complete 2023-03-08T02:46:43.5559979Z 8c656d07c114: Verifying Checksum 2023-03-08T02:46:43.5560703Z 8c656d07c114: Download complete 2023-03-08T02:46:43.9326684Z 86320342bcd0: Verifying Checksum 2023-03-08T02:46:43.9327189Z 86320342bcd0: Download complete 2023-03-08T02:46:44.2368400Z 559c0a582918: Download complete 2023-03-08T02:46:44.2598651Z f4b9815ad375: Verifying Checksum 2023-03-08T02:46:44.2599188Z f4b9815ad375: Download complete 2023-03-08T02:46:44.8614116Z ba972896ba4a: Download complete 2023-03-08T02:46:45.1602571Z 3c800a976df4: Verifying Checksum 2023-03-08T02:46:45.1602950Z 3c800a976df4: Download complete 2023-03-08T02:46:45.2040756Z ea3a51f74b8e: Verifying Checksum 2023-03-08T02:46:45.2041283Z ea3a51f74b8e: Download complete 2023-03-08T02:46:45.4937430Z 810cfd043ef3: Verifying Checksum 2023-03-08T02:46:45.4937987Z 810cfd043ef3: Download complete 2023-03-08T02:46:45.8207686Z 7145aecfb4a6: Verifying Checksum 2023-03-08T02:46:45.8208277Z 7145aecfb4a6: Download complete 2023-03-08T02:46:46.3974712Z 34a279bd458a: Download complete 2023-03-08T02:46:46.6893922Z 2454956ec55e: Verifying Checksum 2023-03-08T02:46:46.6894379Z 2454956ec55e: Download complete 2023-03-08T02:46:47.0400166Z 3b5185ebd5e7: Download complete 2023-03-08T02:46:47.3598070Z cdcd60c75136: Verifying Checksum 2023-03-08T02:46:47.3598621Z cdcd60c75136: Download complete 2023-03-08T02:46:47.6659046Z 898f0f4942b4: Verifying Checksum 2023-03-08T02:46:47.6659566Z 898f0f4942b4: Download complete 2023-03-08T02:46:48.2889127Z 784bb0b58a0d: Verifying Checksum 2023-03-08T02:46:48.2889658Z 784bb0b58a0d: Download complete 2023-03-08T02:46:48.6341733Z a7d3b22585a1: Verifying Checksum 2023-03-08T02:46:48.6342250Z a7d3b22585a1: Download complete 2023-03-08T02:46:49.7918619Z b916fdfdfcb5: Verifying Checksum 2023-03-08T02:46:49.7919015Z b916fdfdfcb5: Download complete 2023-03-08T02:46:50.0993567Z b85022ab5920: Verifying Checksum 2023-03-08T02:46:50.0994133Z b85022ab5920: Download complete 2023-03-08T02:46:54.2928882Z 1aece8b4bd63: Pull complete 2023-03-08T02:46:54.3647085Z 8030ee497010: Pull complete 2023-03-08T02:46:54.4282383Z a41332b1bef3: Pull complete 2023-03-08T02:46:54.5143962Z aed8824e0575: Pull complete 2023-03-08T02:47:21.1329718Z ecf188d26601: Pull complete 2023-03-08T02:47:21.2502270Z 75558b4a699c: Verifying Checksum 2023-03-08T02:47:21.2505253Z 75558b4a699c: Download complete 2023-03-08T02:47:21.6590757Z 8c74646e5134: Pull complete 2023-03-08T02:47:21.6602099Z 7818bcc94020: Verifying Checksum 2023-03-08T02:47:21.6604420Z 7818bcc94020: Download complete 2023-03-08T02:47:22.1844411Z cb9c93f25a34: Pull complete 2023-03-08T02:47:27.4586720Z 8e9933b60756: Pull complete 2023-03-08T02:47:27.5152937Z 74d24d3330f0: Pull complete 2023-03-08T02:47:27.5744868Z a90c1a192b83: Pull complete 2023-03-08T02:47:27.6313009Z 8fb22cf02bcd: Pull complete 2023-03-08T02:47:29.3460176Z 92854560e3ba: Pull complete 2023-03-08T02:47:29.4048065Z 444c90fcb6b3: Pull complete 2023-03-08T02:47:29.4665109Z 849d613e1f4f: Pull complete 2023-03-08T02:47:29.5216322Z 09249daa7056: Pull complete 2023-03-08T02:48:03.5354298Z 75558b4a699c: Pull complete 2023-03-08T02:48:04.0393210Z bbe1302c0658: Pull complete 2023-03-08T02:48:04.5554847Z 3ddb6b2ea880: Pull complete 2023-03-08T02:48:05.0169859Z e0726beb3694: Pull complete 2023-03-08T02:48:05.5189511Z b5a9a355bdf8: Pull complete 2023-03-08T02:48:05.9930415Z b5b2af6db879: Pull complete 2023-03-08T02:48:08.4537342Z ebb48cc5f2f8: Pull complete 2023-03-08T02:48:08.5181326Z 4640a3951080: Pull complete 2023-03-08T02:48:08.5735280Z 42168f59750f: Pull complete 2023-03-08T02:48:08.6655560Z d05e3de10596: Pull complete 2023-03-08T02:48:08.7197619Z 5849c6bb9570: Pull complete 2023-03-08T02:48:08.7790937Z 8dfe045dbd77: Pull complete 2023-03-08T02:48:09.9020029Z 5b2adf07ae20: Verifying Checksum 2023-03-08T02:48:09.9020579Z 5b2adf07ae20: Download complete 2023-03-08T02:48:14.0551509Z 86320342bcd0: Pull complete 2023-03-08T02:48:14.5271091Z f40bd470d742: Pull complete 2023-03-08T02:48:15.0034921Z 67b32c0fb809: Pull complete 2023-03-08T02:48:15.8754029Z 585dfa7633d3: Pull complete 2023-03-08T02:48:15.9297949Z 5392ad3482eb: Pull complete 2023-03-08T02:48:15.9938298Z 8c656d07c114: Pull complete 2023-03-08T02:48:16.3104483Z f4b9815ad375: Pull complete 2023-03-08T02:48:16.3662598Z 559c0a582918: Pull complete 2023-03-08T02:48:17.6961271Z ea3a51f74b8e: Pull complete 2023-03-08T02:48:17.7526506Z a5576526a324: Pull complete 2023-03-08T02:48:17.8103384Z ba972896ba4a: Pull complete 2023-03-08T02:48:17.9204893Z 3c800a976df4: Pull complete 2023-03-08T02:48:36.4279854Z 5537b4d41660: Verifying Checksum 2023-03-08T02:48:36.4280342Z 5537b4d41660: Download complete 2023-03-08T02:49:26.4693612Z 5537b4d41660: Pull complete 2023-03-08T02:49:26.5267025Z 810cfd043ef3: Pull complete 2023-03-08T02:49:26.5838555Z 7145aecfb4a6: Pull complete 2023-03-08T02:49:26.9867478Z 34a279bd458a: Pull complete 2023-03-08T02:49:27.4756791Z 2454956ec55e: Pull complete 2023-03-08T02:49:28.1453773Z 3b5185ebd5e7: Pull complete 2023-03-08T02:49:28.7459959Z cdcd60c75136: Pull complete 2023-03-08T02:49:29.0326857Z 898f0f4942b4: Pull complete 2023-03-08T02:49:29.6997883Z 784bb0b58a0d: Pull complete 2023-03-08T02:49:29.7558105Z a7d3b22585a1: Pull complete 2023-03-08T02:49:31.6079275Z b916fdfdfcb5: Pull complete 2023-03-08T02:49:31.6638017Z b85022ab5920: Pull complete 2023-03-08T02:50:07.7110012Z 5b2adf07ae20: Pull complete 2023-03-08T02:50:07.7688341Z 7818bcc94020: Pull complete 2023-03-08T02:50:07.7781003Z Digest: sha256:165d30726a91aa6e69dd6f42594716968641616ae08d6a929d835657099a4f8a 2023-03-08T02:50:07.7815361Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:50:07.7841724Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:50:07.8032841Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2023-03-08T02:50:07.8033127Z with: 2023-03-08T02:50:07.8033336Z driver-version: 515.76 2023-03-08T02:50:07.8033552Z env: 2023-03-08T02:50:07.8033748Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:07.8033986Z ##[endgroup] 2023-03-08T02:50:07.8066148Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-03-08T02:50:07.8066437Z with: 2023-03-08T02:50:07.8066630Z timeout_minutes: 10 2023-03-08T02:50:07.8066854Z max_attempts: 3 2023-03-08T02:50:07.8074075Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}" 2023-03-08T02:50:07.8081159Z retry_wait_seconds: 10 2023-03-08T02:50:07.8081411Z polling_interval_seconds: 1 2023-03-08T02:50:07.8081658Z warning_on_retry: true 2023-03-08T02:50:07.8081880Z continue_on_error: false 2023-03-08T02:50:07.8082096Z env: 2023-03-08T02:50:07.8082305Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:07.8082530Z DRIVER_VERSION: 515.76 2023-03-08T02:50:07.8082754Z ##[endgroup] 2023-03-08T02:50:07.8702102Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run == 2023-03-08T02:50:07.8702957Z + install_nvidia_driver_common 2023-03-08T02:50:07.8705336Z + echo 'Before installing NVIDIA driver' 2023-03-08T02:50:07.8705733Z + lspci 2023-03-08T02:50:07.8706926Z Before installing NVIDIA driver 2023-03-08T02:50:07.8820554Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-03-08T02:50:07.8821160Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03) 2023-03-08T02:50:07.8821615Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 2023-03-08T02:50:07.8822477Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI 2023-03-08T02:50:07.8823336Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1) 2023-03-08T02:50:07.8823919Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device 2023-03-08T02:50:07.8824491Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG 2023-03-08T02:50:07.8824931Z + lsmod 2023-03-08T02:50:07.8845846Z Module Size Used by 2023-03-08T02:50:07.8846269Z nvidia_modeset 1146880 0 2023-03-08T02:50:07.8846666Z nvidia_uvm 1331200 0 2023-03-08T02:50:07.8846910Z veth 32768 0 2023-03-08T02:50:07.8847139Z xt_conntrack 16384 1 2023-03-08T02:50:07.8852024Z xt_MASQUERADE 20480 1 2023-03-08T02:50:07.8852538Z xfrm_user 40960 1 2023-03-08T02:50:07.8853017Z xfrm_algo 16384 1 xfrm_user 2023-03-08T02:50:07.8853436Z xt_addrtype 16384 2 2023-03-08T02:50:07.8853830Z iptable_filter 16384 1 2023-03-08T02:50:07.8854177Z iptable_nat 16384 1 2023-03-08T02:50:07.8854462Z nf_nat 49152 2 iptable_nat,xt_MASQUERADE 2023-03-08T02:50:07.8854712Z bpfilter 16384 0 2023-03-08T02:50:07.8855093Z br_netfilter 28672 0 2023-03-08T02:50:07.8855491Z bridge 307200 1 br_netfilter 2023-03-08T02:50:07.8855859Z stp 16384 1 bridge 2023-03-08T02:50:07.8856269Z llc 16384 2 bridge,stp 2023-03-08T02:50:07.8856764Z aufs 270336 0 2023-03-08T02:50:07.8857181Z overlay 151552 0 2023-03-08T02:50:07.8857559Z nls_iso8859_1 16384 1 2023-03-08T02:50:07.8857861Z dm_multipath 40960 0 2023-03-08T02:50:07.8858184Z scsi_dh_rdac 16384 0 2023-03-08T02:50:07.8858585Z scsi_dh_emc 16384 0 2023-03-08T02:50:07.8859050Z scsi_dh_alua 20480 0 2023-03-08T02:50:07.8859534Z nvidia 40837120 2 nvidia_uvm,nvidia_modeset 2023-03-08T02:50:07.8859862Z binfmt_misc 24576 1 2023-03-08T02:50:07.8860104Z crct10dif_pclmul 16384 1 2023-03-08T02:50:07.8860342Z psmouse 180224 0 2023-03-08T02:50:07.8860554Z crc32_pclmul 16384 0 2023-03-08T02:50:07.8860784Z ghash_clmulni_intel 16384 0 2023-03-08T02:50:07.8861017Z input_leds 16384 0 2023-03-08T02:50:07.8861429Z aesni_intel 376832 0 2023-03-08T02:50:07.8861653Z virtio_net 61440 0 2023-03-08T02:50:07.8861879Z serio_raw 20480 0 2023-03-08T02:50:07.8862107Z net_failover 20480 1 virtio_net 2023-03-08T02:50:07.8862370Z failover 16384 1 net_failover 2023-03-08T02:50:07.8862632Z crypto_simd 16384 1 aesni_intel 2023-03-08T02:50:07.8862908Z cryptd 24576 2 crypto_simd,ghash_clmulni_intel 2023-03-08T02:50:07.8863179Z efi_pstore 16384 0 2023-03-08T02:50:07.8863405Z sch_fq_codel 24576 13 2023-03-08T02:50:07.8863626Z drm 618496 1 nvidia 2023-03-08T02:50:07.8863870Z virtio_rng 16384 0 2023-03-08T02:50:07.8864148Z ip_tables 32768 2 iptable_filter,iptable_nat 2023-03-08T02:50:07.8864536Z x_tables 53248 6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE 2023-03-08T02:50:07.8864855Z autofs4 49152 2 2023-03-08T02:50:07.8865080Z + modinfo nvidia 2023-03-08T02:50:07.8865575Z filename: /lib/modules/5.15.0-1030-gcp/kernel/drivers/video/nvidia.ko 2023-03-08T02:50:07.8866064Z firmware: nvidia/515.76/gsp.bin 2023-03-08T02:50:07.8866668Z alias: char-major-195-* 2023-03-08T02:50:07.8867251Z version: 515.76 2023-03-08T02:50:07.8867900Z supported: external 2023-03-08T02:50:07.8868236Z license: NVIDIA 2023-03-08T02:50:07.8868485Z srcversion: 9B058442E3B98BD738AB1CF 2023-03-08T02:50:07.8868771Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-03-08T02:50:07.8869042Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-03-08T02:50:07.8869322Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-03-08T02:50:07.8869716Z depends: drm 2023-03-08T02:50:07.8869930Z retpoline: Y 2023-03-08T02:50:07.8870170Z name: nvidia 2023-03-08T02:50:07.8870710Z vermagic: 5.15.0-1030-gcp SMP mod_unload modversions 2023-03-08T02:50:07.8871206Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-03-08T02:50:07.8871833Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-03-08T02:50:07.8872448Z parm: NVreg_ResmanDebugLevel:int 2023-03-08T02:50:07.8872939Z parm: NVreg_RmLogonRC:int 2023-03-08T02:50:07.8873423Z parm: NVreg_ModifyDeviceFiles:int 2023-03-08T02:50:07.8873922Z parm: NVreg_DeviceFileUID:int 2023-03-08T02:50:07.8874298Z parm: NVreg_DeviceFileGID:int 2023-03-08T02:50:07.8874671Z parm: NVreg_DeviceFileMode:int 2023-03-08T02:50:07.8875303Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-03-08T02:50:07.8875997Z parm: NVreg_UsePageAttributeTable:int 2023-03-08T02:50:07.8876428Z parm: NVreg_EnablePCIeGen3:int 2023-03-08T02:50:07.8876694Z parm: NVreg_EnableMSI:int 2023-03-08T02:50:07.8876955Z parm: NVreg_TCEBypassMode:int 2023-03-08T02:50:07.8877226Z parm: NVreg_EnableStreamMemOPs:int 2023-03-08T02:50:07.8877560Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-03-08T02:50:07.8877977Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-03-08T02:50:07.8878307Z parm: NVreg_EnableS0ixPowerManagement:int 2023-03-08T02:50:07.8878805Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-03-08T02:50:07.8879524Z parm: NVreg_DynamicPowerManagement:int 2023-03-08T02:50:07.8879957Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-03-08T02:50:07.8880305Z parm: NVreg_EnableGpuFirmware:int 2023-03-08T02:50:07.8880604Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-03-08T02:50:07.8880935Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-03-08T02:50:07.8881260Z parm: NVreg_EnableUserNUMAManagement:int 2023-03-08T02:50:07.8881555Z parm: NVreg_MemoryPoolSize:int 2023-03-08T02:50:07.8881839Z parm: NVreg_KMallocHeapMaxSize:int 2023-03-08T02:50:07.8882119Z parm: NVreg_VMallocHeapMaxSize:int 2023-03-08T02:50:07.8882543Z parm: NVreg_IgnoreMMIOCheck:int 2023-03-08T02:50:07.8882816Z parm: NVreg_NvLinkDisable:int 2023-03-08T02:50:07.8883130Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-03-08T02:50:07.8883439Z parm: NVreg_RegisterPCIDriver:int 2023-03-08T02:50:07.8883733Z parm: NVreg_EnableDbgBreakpoint:int 2023-03-08T02:50:07.8884021Z parm: NVreg_RegistryDwords:charp 2023-03-08T02:50:07.8884313Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-03-08T02:50:07.8884597Z parm: NVreg_RmMsg:charp 2023-03-08T02:50:07.8884860Z parm: NVreg_GpuBlacklist:charp 2023-03-08T02:50:07.8885135Z parm: NVreg_TemporaryFilePath:charp 2023-03-08T02:50:07.8885419Z parm: NVreg_ExcludedGpus:charp 2023-03-08T02:50:07.8885698Z parm: NVreg_DmaRemapPeerMmio:int 2023-03-08T02:50:07.8885958Z parm: rm_firmware_active:charp 2023-03-08T02:50:07.8886205Z + HAS_NVIDIA_DRIVER=0 2023-03-08T02:50:07.8886544Z ++ command -v nvidia-smi 2023-03-08T02:50:07.8886832Z + '[' -x /usr/bin/nvidia-smi ']' 2023-03-08T02:50:07.8887057Z + set +e 2023-03-08T02:50:07.8887440Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2023-03-08T02:50:10.7755144Z + INSTALLED_DRIVER_VERSION=515.76 2023-03-08T02:50:10.7755567Z + NVIDIA_SMI_STATUS=0 2023-03-08T02:50:10.7756357Z + '[' 0 -ne 0 ']' 2023-03-08T02:50:10.7756808Z + '[' 515.76 '!=' 515.76 ']' 2023-03-08T02:50:10.7757079Z + HAS_NVIDIA_DRIVER=1 2023-03-08T02:50:10.7757549Z + echo 'NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation' 2023-03-08T02:50:10.7757872Z + set -e 2023-03-08T02:50:10.7758118Z + '[' 1 -eq 0 ']' 2023-03-08T02:50:10.7758750Z NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation 2023-03-08T02:50:10.7759095Z + post_install_nvidia_driver_common 2023-03-08T02:50:10.7762007Z + sudo modprobe nvidia 2023-03-08T02:50:10.7881735Z + echo 'After installing NVIDIA driver' 2023-03-08T02:50:10.7882070Z + lspci 2023-03-08T02:50:10.7882292Z After installing NVIDIA driver 2023-03-08T02:50:10.8000414Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-03-08T02:50:10.8001173Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03) 2023-03-08T02:50:10.8001537Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 2023-03-08T02:50:10.8002002Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI 2023-03-08T02:50:10.8002337Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1) 2023-03-08T02:50:10.8002735Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device 2023-03-08T02:50:10.8003073Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG 2023-03-08T02:50:10.8003331Z + lsmod 2023-03-08T02:50:10.8022839Z Module Size Used by 2023-03-08T02:50:10.8023156Z nvidia_modeset 1146880 0 2023-03-08T02:50:10.8023543Z nvidia_uvm 1331200 0 2023-03-08T02:50:10.8023917Z veth 32768 0 2023-03-08T02:50:10.8024302Z xt_conntrack 16384 1 2023-03-08T02:50:10.8024740Z xt_MASQUERADE 20480 1 2023-03-08T02:50:10.8025164Z xfrm_user 40960 1 2023-03-08T02:50:10.8025603Z xfrm_algo 16384 1 xfrm_user 2023-03-08T02:50:10.8025993Z xt_addrtype 16384 2 2023-03-08T02:50:10.8026393Z iptable_filter 16384 1 2023-03-08T02:50:10.8026780Z iptable_nat 16384 1 2023-03-08T02:50:10.8027376Z nf_nat 49152 2 iptable_nat,xt_MASQUERADE 2023-03-08T02:50:10.8027818Z bpfilter 16384 0 2023-03-08T02:50:10.8028298Z br_netfilter 28672 0 2023-03-08T02:50:10.8028712Z bridge 307200 1 br_netfilter 2023-03-08T02:50:10.8029230Z stp 16384 1 bridge 2023-03-08T02:50:10.8029595Z llc 16384 2 bridge,stp 2023-03-08T02:50:10.8029820Z aufs 270336 0 2023-03-08T02:50:10.8030053Z overlay 151552 0 2023-03-08T02:50:10.8030536Z nls_iso8859_1 16384 1 2023-03-08T02:50:10.8030758Z dm_multipath 40960 0 2023-03-08T02:50:10.8031019Z scsi_dh_rdac 16384 0 2023-03-08T02:50:10.8031245Z scsi_dh_emc 16384 0 2023-03-08T02:50:10.8031475Z scsi_dh_alua 20480 0 2023-03-08T02:50:10.8031734Z nvidia 40837120 2 nvidia_uvm,nvidia_modeset 2023-03-08T02:50:10.8032000Z binfmt_misc 24576 1 2023-03-08T02:50:10.8032242Z crct10dif_pclmul 16384 1 2023-03-08T02:50:10.8032534Z psmouse 180224 0 2023-03-08T02:50:10.8032769Z crc32_pclmul 16384 0 2023-03-08T02:50:10.8033012Z ghash_clmulni_intel 16384 0 2023-03-08T02:50:10.8033240Z input_leds 16384 0 2023-03-08T02:50:10.8033473Z aesni_intel 376832 0 2023-03-08T02:50:10.8033703Z virtio_net 61440 0 2023-03-08T02:50:10.8033910Z serio_raw 20480 0 2023-03-08T02:50:10.8034150Z net_failover 20480 1 virtio_net 2023-03-08T02:50:10.8034417Z failover 16384 1 net_failover 2023-03-08T02:50:10.8034669Z crypto_simd 16384 1 aesni_intel 2023-03-08T02:50:10.8034962Z cryptd 24576 2 crypto_simd,ghash_clmulni_intel 2023-03-08T02:50:10.8035234Z efi_pstore 16384 0 2023-03-08T02:50:10.8035455Z sch_fq_codel 24576 13 2023-03-08T02:50:10.8035688Z drm 618496 1 nvidia 2023-03-08T02:50:10.8035921Z virtio_rng 16384 0 2023-03-08T02:50:10.8036183Z ip_tables 32768 2 iptable_filter,iptable_nat 2023-03-08T02:50:10.8036570Z x_tables 53248 6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE 2023-03-08T02:50:10.8036910Z autofs4 49152 2 2023-03-08T02:50:10.8037222Z + modinfo nvidia 2023-03-08T02:50:10.8040263Z filename: /lib/modules/5.15.0-1030-gcp/kernel/drivers/video/nvidia.ko 2023-03-08T02:50:10.8040842Z firmware: nvidia/515.76/gsp.bin 2023-03-08T02:50:10.8041368Z alias: char-major-195-* 2023-03-08T02:50:10.8041818Z version: 515.76 2023-03-08T02:50:10.8042264Z supported: external 2023-03-08T02:50:10.8042746Z license: NVIDIA 2023-03-08T02:50:10.8043007Z srcversion: 9B058442E3B98BD738AB1CF 2023-03-08T02:50:10.8043294Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-03-08T02:50:10.8043585Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-03-08T02:50:10.8043870Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-03-08T02:50:10.8044106Z depends: drm 2023-03-08T02:50:10.8044327Z retpoline: Y 2023-03-08T02:50:10.8044547Z name: nvidia 2023-03-08T02:50:10.8044899Z vermagic: 5.15.0-1030-gcp SMP mod_unload modversions 2023-03-08T02:50:10.8045238Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-03-08T02:50:10.8045602Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-03-08T02:50:10.8045922Z parm: NVreg_ResmanDebugLevel:int 2023-03-08T02:50:10.8046190Z parm: NVreg_RmLogonRC:int 2023-03-08T02:50:10.8046508Z parm: NVreg_ModifyDeviceFiles:int 2023-03-08T02:50:10.8046785Z parm: NVreg_DeviceFileUID:int 2023-03-08T02:50:10.8047037Z parm: NVreg_DeviceFileGID:int 2023-03-08T02:50:10.8047305Z parm: NVreg_DeviceFileMode:int 2023-03-08T02:50:10.8047639Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-03-08T02:50:10.8047972Z parm: NVreg_UsePageAttributeTable:int 2023-03-08T02:50:10.8048264Z parm: NVreg_EnablePCIeGen3:int 2023-03-08T02:50:10.8048531Z parm: NVreg_EnableMSI:int 2023-03-08T02:50:10.8048781Z parm: NVreg_TCEBypassMode:int 2023-03-08T02:50:10.8049070Z parm: NVreg_EnableStreamMemOPs:int 2023-03-08T02:50:10.8049406Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-03-08T02:50:10.8049752Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-03-08T02:50:10.8050092Z parm: NVreg_EnableS0ixPowerManagement:int 2023-03-08T02:50:10.8050634Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-03-08T02:50:10.8051000Z parm: NVreg_DynamicPowerManagement:int 2023-03-08T02:50:10.8051366Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-03-08T02:50:10.8051734Z parm: NVreg_EnableGpuFirmware:int 2023-03-08T02:50:10.8052038Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-03-08T02:50:10.8052356Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-03-08T02:50:10.8052749Z parm: NVreg_EnableUserNUMAManagement:int 2023-03-08T02:50:10.8053050Z parm: NVreg_MemoryPoolSize:int 2023-03-08T02:50:10.8053324Z parm: NVreg_KMallocHeapMaxSize:int 2023-03-08T02:50:10.8053625Z parm: NVreg_VMallocHeapMaxSize:int 2023-03-08T02:50:10.8053912Z parm: NVreg_IgnoreMMIOCheck:int 2023-03-08T02:50:10.8054172Z parm: NVreg_NvLinkDisable:int 2023-03-08T02:50:10.8054491Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-03-08T02:50:10.8054815Z parm: NVreg_RegisterPCIDriver:int 2023-03-08T02:50:10.8055096Z parm: NVreg_EnableDbgBreakpoint:int 2023-03-08T02:50:10.8055389Z parm: NVreg_RegistryDwords:charp 2023-03-08T02:50:10.8055694Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-03-08T02:50:10.8055977Z parm: NVreg_RmMsg:charp 2023-03-08T02:50:10.8056225Z parm: NVreg_GpuBlacklist:charp 2023-03-08T02:50:10.8056509Z parm: NVreg_TemporaryFilePath:charp 2023-03-08T02:50:10.8056787Z parm: NVreg_ExcludedGpus:charp 2023-03-08T02:50:10.8057054Z parm: NVreg_DmaRemapPeerMmio:int 2023-03-08T02:50:10.8057327Z parm: rm_firmware_active:charp 2023-03-08T02:50:10.8057556Z + set +e 2023-03-08T02:50:10.8057865Z + nvidia-smi 2023-03-08T02:50:13.1854798Z Wed Mar 8 02:50:13 2023 2023-03-08T02:50:13.1856235Z +-----------------------------------------------------------------------------+ 2023-03-08T02:50:13.1856753Z | NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 | 2023-03-08T02:50:13.1857250Z |-------------------------------+----------------------+----------------------+ 2023-03-08T02:50:13.1857838Z | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-03-08T02:50:13.1858329Z | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2023-03-08T02:50:13.1858664Z | | | MIG M. | 2023-03-08T02:50:13.1858941Z |===============================+======================+======================| 2023-03-08T02:50:13.2429249Z | 0 NVIDIA A100-SXM... Off | 00000000:00:04.0 Off | 0 | 2023-03-08T02:50:13.2429834Z | N/A 32C P0 53W / 400W | 0MiB / 40960MiB | 2% Default | 2023-03-08T02:50:13.2430146Z | | | Disabled | 2023-03-08T02:50:13.2430600Z +-------------------------------+----------------------+----------------------+ 2023-03-08T02:50:13.2430951Z 2023-03-08T02:50:13.2431379Z +-----------------------------------------------------------------------------+ 2023-03-08T02:50:13.2431735Z | Processes: | 2023-03-08T02:50:13.2432042Z | GPU GI CI PID Type Process name GPU Memory | 2023-03-08T02:50:13.2432356Z | ID ID Usage | 2023-03-08T02:50:13.2432635Z |=============================================================================| 2023-03-08T02:50:13.2432937Z | No running processes found | 2023-03-08T02:50:13.2433456Z +-----------------------------------------------------------------------------+ 2023-03-08T02:50:13.9250975Z + NVIDIA_SMI_STATUS=0 2023-03-08T02:50:13.9251610Z + '[' 0 -eq 0 ']' 2023-03-08T02:50:13.9251941Z + echo 'INFO: Ignoring allowed status 0' 2023-03-08T02:50:13.9252494Z + set -e 2023-03-08T02:50:13.9252706Z INFO: Ignoring allowed status 0 2023-03-08T02:50:13.9259396Z == Installing nvidia container toolkit for ubuntu20.04 == 2023-03-08T02:50:13.9262913Z + sudo apt-get install -y nvidia-docker2 2023-03-08T02:50:13.9932806Z Reading package lists... 2023-03-08T02:50:14.1723644Z Building dependency tree... 2023-03-08T02:50:14.1730269Z Reading state information... 2023-03-08T02:50:14.3293129Z nvidia-docker2 is already the newest version (2.12.0-1). 2023-03-08T02:50:14.3293844Z The following packages were automatically installed and are no longer required: 2023-03-08T02:50:14.3294669Z libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2 2023-03-08T02:50:14.3295469Z libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2 2023-03-08T02:50:14.3296233Z libexpat1-dev libmm-glib0 libnspr4 libnss3 libnuma1 libparted-fs-resize0 2023-03-08T02:50:14.3299846Z libpython3-dev libpython3.8-dev libudisks2-0 usb-modeswitch 2023-03-08T02:50:14.3300407Z usb-modeswitch-data 2023-03-08T02:50:14.3300892Z Use 'sudo apt autoremove' to remove them. 2023-03-08T02:50:14.4215265Z 0 upgraded, 0 newly installed, 0 to remove and 41 not upgraded. 2023-03-08T02:50:14.5010682Z + sudo systemctl restart docker 2023-03-08T02:50:22.8863302Z Command completed after 1 attempt(s). 2023-03-08T02:50:22.8927857Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-03-08T02:50:22.8928303Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-03-08T02:50:22.8928646Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2023-03-08T02:50:22.8928999Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2023-03-08T02:50:22.8946986Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:50:22.8947546Z env: 2023-03-08T02:50:22.8947767Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:22.8947998Z GPU_FLAG: --gpus all 2023-03-08T02:50:22.8948222Z ##[endgroup] 2023-03-08T02:50:24.0161995Z Requirement already satisfied: psutil==5.9.1 in /home/ubuntu/.local/lib/python3.8/site-packages (5.9.1) 2023-03-08T02:50:24.0243898Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ubuntu/.local/lib/python3.8/site-packages (11.525.84) 2023-03-08T02:50:24.2329857Z Prepare all required actions 2023-03-08T02:50:24.2330464Z Getting action download info 2023-03-08T02:50:24.4455095Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2023-03-08T02:50:26.6453448Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2023-03-08T02:50:26.9986989Z ##[group]Run ./.github/actions/download-build-artifacts 2023-03-08T02:50:26.9987589Z with: 2023-03-08T02:50:26.9987927Z name: linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:50:26.9988196Z env: 2023-03-08T02:50:26.9988415Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:26.9988643Z GPU_FLAG: --gpus all 2023-03-08T02:50:26.9988866Z ##[endgroup] 2023-03-08T02:50:27.0017252Z ##[group]Run seemethere/download-artifact-s3@v4 2023-03-08T02:50:27.0017503Z with: 2023-03-08T02:50:27.0017769Z name: linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:50:27.0018058Z s3-bucket: gha-artifacts 2023-03-08T02:50:27.0018318Z region: us-east-1 2023-03-08T02:50:27.0018522Z env: 2023-03-08T02:50:27.0018720Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:27.0018958Z GPU_FLAG: --gpus all 2023-03-08T02:50:27.0019173Z ##[endgroup] 2023-03-08T02:50:27.5864818Z Found 1 objects with prefix pytorch/pytorch/4360459833/linux-bionic-cuda11.8-py3.10-gcc7-sm80/ 2023-03-08T02:50:27.5865456Z Starting download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-03-08T02:50:34.8528799Z Finished download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-03-08T02:50:34.8529143Z 2023-03-08T02:50:34.8552288Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2023-03-08T02:50:34.8562602Z Artifact download has finished successfully 2023-03-08T02:50:34.8731434Z ##[group]Run unzip -o artifacts.zip 2023-03-08T02:50:34.8731730Z unzip -o artifacts.zip 2023-03-08T02:50:34.8750329Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:50:34.8750614Z env: 2023-03-08T02:50:34.8750843Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:34.8751080Z GPU_FLAG: --gpus all 2023-03-08T02:50:34.8751312Z ##[endgroup] 2023-03-08T02:50:34.8806842Z Archive: artifacts.zip 2023-03-08T02:50:34.8808699Z creating: dist/ 2023-03-08T02:50:36.8646867Z inflating: dist/torch-2.1.0a0+gitc88aa33-cp310-cp310-linux_x86_64.whl 2023-03-08T02:50:36.8647452Z creating: build/custom_test_artifacts/ 2023-03-08T02:50:36.8647980Z creating: build/custom_test_artifacts/custom-op-build/ 2023-03-08T02:50:36.8648443Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2023-03-08T02:50:36.8653533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2023-03-08T02:50:36.8654072Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2023-03-08T02:50:36.8654643Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-03-08T02:50:36.8655187Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-03-08T02:50:36.8655724Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-03-08T02:50:36.8657372Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-03-08T02:50:36.8658302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-03-08T02:50:36.8658858Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-03-08T02:50:36.8659399Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-03-08T02:50:36.8661578Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-03-08T02:50:36.8662805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-03-08T02:50:36.8663797Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-03-08T02:50:36.8664385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-03-08T02:50:36.8665850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-03-08T02:50:36.8666465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-03-08T02:50:36.8667038Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-03-08T02:50:36.8667816Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-03-08T02:50:36.8722212Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-03-08T02:50:36.8722987Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-03-08T02:50:36.8723705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-03-08T02:50:36.8724411Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-03-08T02:50:36.8725115Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-03-08T02:50:36.8725794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-03-08T02:50:36.8726705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-03-08T02:50:36.8727372Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-03-08T02:50:36.8728061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-03-08T02:50:36.8772676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-03-08T02:50:36.8819654Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-03-08T02:50:36.8820361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-03-08T02:50:36.8821010Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-03-08T02:50:36.8821644Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-03-08T02:50:36.8822260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-03-08T02:50:36.8822929Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-03-08T02:50:36.8823535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-03-08T02:50:36.8824442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-03-08T02:50:36.8896346Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-03-08T02:50:36.8968364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-03-08T02:50:36.8969011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-03-08T02:50:36.8969784Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2023-03-08T02:50:36.8970325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2023-03-08T02:50:36.8970842Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2023-03-08T02:50:36.8971368Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2023-03-08T02:50:36.8971932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2023-03-08T02:50:36.8972600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2023-03-08T02:50:36.8973167Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2023-03-08T02:50:36.8973729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2023-03-08T02:50:36.8974308Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2023-03-08T02:50:36.8974880Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2023-03-08T02:50:36.8975441Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2023-03-08T02:50:36.8976009Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2023-03-08T02:50:36.8976575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2023-03-08T02:50:36.8997566Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2023-03-08T02:50:36.9114847Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2023-03-08T02:50:36.9115546Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2023-03-08T02:50:36.9116372Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2023-03-08T02:50:36.9117005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2023-03-08T02:50:36.9117591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2023-03-08T02:50:36.9118174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2023-03-08T02:50:36.9118766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2023-03-08T02:50:36.9119351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2023-03-08T02:50:36.9119930Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2023-03-08T02:50:36.9120520Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2023-03-08T02:50:36.9121108Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2023-03-08T02:50:36.9141790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2023-03-08T02:50:36.9228729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2023-03-08T02:50:36.9229373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-03-08T02:50:36.9229968Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2023-03-08T02:50:36.9230523Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2023-03-08T02:50:36.9231031Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2023-03-08T02:50:36.9231919Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2023-03-08T02:50:36.9232435Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2023-03-08T02:50:36.9235800Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2023-03-08T02:50:36.9236275Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2023-03-08T02:50:36.9236939Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2023-03-08T02:50:36.9330865Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2023-03-08T02:50:36.9395357Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2023-03-08T02:50:36.9395803Z creating: build/custom_test_artifacts/jit-hook-build/ 2023-03-08T02:50:36.9396247Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2023-03-08T02:50:36.9402836Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2023-03-08T02:50:36.9403361Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2023-03-08T02:50:36.9403878Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-03-08T02:50:36.9404421Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-03-08T02:50:36.9404944Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-03-08T02:50:36.9406982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-03-08T02:50:36.9409198Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-03-08T02:50:36.9409793Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-03-08T02:50:36.9410351Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-03-08T02:50:36.9411311Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-03-08T02:50:36.9412478Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-03-08T02:50:36.9413931Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-03-08T02:50:36.9414766Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-03-08T02:50:36.9415637Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-03-08T02:50:36.9416255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-03-08T02:50:36.9416811Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-03-08T02:50:36.9417341Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-03-08T02:50:36.9471814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-03-08T02:50:36.9472543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-03-08T02:50:36.9473289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-03-08T02:50:36.9474011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-03-08T02:50:36.9474692Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-03-08T02:50:36.9475366Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-03-08T02:50:36.9476036Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-03-08T02:50:36.9476703Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-03-08T02:50:36.9477526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-03-08T02:50:36.9522130Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-03-08T02:50:36.9568651Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-03-08T02:50:36.9569452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-03-08T02:50:36.9570672Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-03-08T02:50:36.9571708Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-03-08T02:50:36.9572904Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-03-08T02:50:36.9573696Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-03-08T02:50:36.9574696Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-03-08T02:50:36.9575314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-03-08T02:50:36.9645857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-03-08T02:50:36.9717658Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-03-08T02:50:36.9718299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-03-08T02:50:36.9718839Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2023-03-08T02:50:36.9719549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2023-03-08T02:50:36.9720063Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2023-03-08T02:50:36.9720591Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2023-03-08T02:50:36.9721172Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2023-03-08T02:50:36.9721757Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2023-03-08T02:50:36.9722336Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2023-03-08T02:50:36.9722963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2023-03-08T02:50:36.9723535Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2023-03-08T02:50:36.9724095Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2023-03-08T02:50:36.9724685Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2023-03-08T02:50:36.9725254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2023-03-08T02:50:36.9725824Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2023-03-08T02:50:36.9747053Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2023-03-08T02:50:36.9818668Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2023-03-08T02:50:36.9819295Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-03-08T02:50:36.9819879Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2023-03-08T02:50:36.9820420Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2023-03-08T02:50:36.9821080Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2023-03-08T02:50:36.9821775Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2023-03-08T02:50:36.9822283Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2023-03-08T02:50:36.9824705Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2023-03-08T02:50:36.9825180Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2023-03-08T02:50:36.9826135Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2023-03-08T02:50:36.9881238Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2023-03-08T02:50:36.9881732Z creating: build/custom_test_artifacts/custom-backend-build/ 2023-03-08T02:50:36.9882211Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2023-03-08T02:50:36.9888413Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2023-03-08T02:50:36.9889230Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2023-03-08T02:50:36.9889801Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-03-08T02:50:36.9890356Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-03-08T02:50:36.9891021Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-03-08T02:50:36.9891965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-03-08T02:50:36.9893090Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-03-08T02:50:36.9893701Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-03-08T02:50:36.9894421Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-03-08T02:50:36.9895686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-03-08T02:50:36.9896859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-03-08T02:50:36.9898335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-03-08T02:50:36.9898946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-03-08T02:50:36.9900099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-03-08T02:50:36.9900970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-03-08T02:50:36.9901570Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-03-08T02:50:36.9902149Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-03-08T02:50:36.9956258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-03-08T02:50:36.9957009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-03-08T02:50:36.9957747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-03-08T02:50:36.9958504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-03-08T02:50:36.9959220Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-03-08T02:50:36.9959923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-03-08T02:50:36.9960776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-03-08T02:50:36.9961486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-03-08T02:50:36.9962173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-03-08T02:50:37.0006818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-03-08T02:50:37.0053809Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-03-08T02:50:37.0054525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-03-08T02:50:37.0055196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-03-08T02:50:37.0055821Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-03-08T02:50:37.0056460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-03-08T02:50:37.0057091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-03-08T02:50:37.0057711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-03-08T02:50:37.0058694Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-03-08T02:50:37.0130921Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-03-08T02:50:37.0202468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-03-08T02:50:37.0203171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-03-08T02:50:37.0203713Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2023-03-08T02:50:37.0204256Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2023-03-08T02:50:37.0204805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2023-03-08T02:50:37.0205354Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2023-03-08T02:50:37.0205940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2023-03-08T02:50:37.0206574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2023-03-08T02:50:37.0207193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2023-03-08T02:50:37.0207811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2023-03-08T02:50:37.0208566Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2023-03-08T02:50:37.0209177Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2023-03-08T02:50:37.0209799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2023-03-08T02:50:37.0210403Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2023-03-08T02:50:37.0210993Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2023-03-08T02:50:37.0214124Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2023-03-08T02:50:37.0364060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2023-03-08T02:50:37.0364760Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2023-03-08T02:50:37.0365370Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2023-03-08T02:50:37.0366018Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2023-03-08T02:50:37.0366651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2023-03-08T02:50:37.0367267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2023-03-08T02:50:37.0367880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2023-03-08T02:50:37.0368506Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2023-03-08T02:50:37.0369136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2023-03-08T02:50:37.0369755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2023-03-08T02:50:37.0370358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2023-03-08T02:50:37.0391488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2023-03-08T02:50:37.0453130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2023-03-08T02:50:37.0454086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-03-08T02:50:37.0454699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2023-03-08T02:50:37.0455270Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2023-03-08T02:50:37.0455817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2023-03-08T02:50:37.0456609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2023-03-08T02:50:37.0457196Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2023-03-08T02:50:37.0459991Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2023-03-08T02:50:37.0461192Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2023-03-08T02:50:37.0461926Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2023-03-08T02:50:37.0581125Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2023-03-08T02:50:37.0628238Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2023-03-08T02:50:37.0628819Z creating: build/lib/ 2023-03-08T02:50:37.0629117Z inflating: build/lib/libclog.a 2023-03-08T02:50:37.0638711Z inflating: build/lib/libpthreadpool.a 2023-03-08T02:50:37.0705773Z inflating: build/lib/libgtest.a 2023-03-08T02:50:37.0808269Z inflating: build/lib/libprotobuf-lite.a 2023-03-08T02:50:37.0901334Z inflating: build/lib/libbenchmark.a 2023-03-08T02:50:37.0909613Z inflating: build/lib/libittnotify.a 2023-03-08T02:50:37.0940376Z inflating: build/lib/libtensorpipe_uv.a 2023-03-08T02:50:37.1457492Z inflating: build/lib/libprotobuf.a 2023-03-08T02:50:37.1593777Z inflating: build/lib/libgloo.a 2023-03-08T02:50:37.1669887Z inflating: build/lib/libasmjit.a 2023-03-08T02:50:37.1691793Z inflating: build/lib/libfmt.a 2023-03-08T02:50:37.1692371Z inflating: build/lib/libfoxi_loader.a 2023-03-08T02:50:37.1693376Z inflating: build/lib/libcaffe2_nvrtc.so 2023-03-08T02:50:37.1775776Z inflating: build/lib/libc10.so 2023-03-08T02:50:37.1776235Z inflating: build/lib/libtorch_global_deps.so 2023-03-08T02:50:37.1785209Z inflating: build/lib/libcpuinfo.a 2023-03-08T02:50:37.2336068Z inflating: build/lib/libprotoc.a 2023-03-08T02:50:37.2344190Z inflating: build/lib/libcpuinfo_internals.a 2023-03-08T02:50:37.2359262Z inflating: build/lib/libqnnpack.a 2023-03-08T02:50:37.2381573Z inflating: build/lib/libpytorch_qnnpack.a 2023-03-08T02:50:37.2383531Z inflating: build/lib/libnnpack_reference_layers.a 2023-03-08T02:50:37.2404836Z inflating: build/lib/libnnpack.a 2023-03-08T02:50:37.2421634Z inflating: build/lib/libgmock.a 2023-03-08T02:50:37.2422285Z inflating: build/lib/libgtest_main.a 2023-03-08T02:50:37.2422814Z inflating: build/lib/libbenchmark_main.a 2023-03-08T02:50:38.1905778Z inflating: build/lib/libdnnl.a 2023-03-08T02:50:38.2539217Z inflating: build/lib/libtensorpipe.a 2023-03-08T02:50:38.2591293Z inflating: build/lib/libc10_cuda.so 2023-03-08T02:50:38.2591835Z inflating: build/lib/libgmock_main.a 2023-03-08T02:50:38.4093161Z inflating: build/lib/libfbgemm.a 2023-03-08T02:50:38.5200220Z inflating: build/lib/libdnnl_graph.a 2023-03-08T02:50:38.5716601Z inflating: build/lib/libkineto.a 2023-03-08T02:50:38.5997197Z inflating: build/lib/libtensorpipe_cuda.a 2023-03-08T02:50:38.6040408Z inflating: build/lib/libcaffe2_protos.a 2023-03-08T02:50:38.6179570Z inflating: build/lib/libXNNPACK.a 2023-03-08T02:50:38.6225851Z inflating: build/lib/libonnx_proto.a 2023-03-08T02:50:38.6940880Z inflating: build/lib/libonnx.a 2023-03-08T02:50:38.7359163Z inflating: build/lib/libgloo_cuda.a 2023-03-08T02:50:41.0751826Z inflating: build/lib/libtorch_cpu.so 2023-03-08T02:50:41.0767978Z inflating: build/lib/libunbox_lib.a 2023-03-08T02:50:42.8358533Z inflating: build/lib/libtorch_cuda.so 2023-03-08T02:50:42.8358982Z inflating: build/lib/libtorch.so 2023-03-08T02:50:43.9431769Z inflating: build/lib/libtorch_cuda_linalg.so 2023-03-08T02:50:43.9432898Z inflating: build/lib/libc10d_cuda_test.so 2023-03-08T02:50:43.9457121Z inflating: build/lib/libjitbackend_test.so 2023-03-08T02:50:43.9515784Z inflating: build/lib/libtorchbind_test.so 2023-03-08T02:50:43.9544984Z inflating: build/lib/libbackend_with_compiler.so 2023-03-08T02:50:43.9549823Z inflating: build/lib/libshm.so 2023-03-08T02:50:44.0250770Z inflating: build/lib/libnvfuser_codegen.so 2023-03-08T02:50:44.1988137Z inflating: build/lib/libtorch_python.so 2023-03-08T02:50:44.2026277Z inflating: build/lib/libnnapi_backend.so 2023-03-08T02:50:44.2026966Z creating: build/bin/ 2023-03-08T02:50:44.2078584Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2023-03-08T02:50:44.2133685Z inflating: build/bin/c10_DeviceGuard_test 2023-03-08T02:50:44.2187389Z inflating: build/bin/c10_Device_test 2023-03-08T02:50:44.2249275Z inflating: build/bin/c10_DispatchKeySet_test 2023-03-08T02:50:44.2299970Z inflating: build/bin/c10_StreamGuard_test 2023-03-08T02:50:44.2352384Z inflating: build/bin/c10_SymInt_test 2023-03-08T02:50:44.2411626Z inflating: build/bin/c10_InlineDeviceGuard_test 2023-03-08T02:50:44.2471405Z inflating: build/bin/c10_InlineStreamGuard_test 2023-03-08T02:50:44.2531467Z inflating: build/bin/c10_SizesAndStrides_test 2023-03-08T02:50:44.2583597Z inflating: build/bin/c10_Array_test 2023-03-08T02:50:44.2640386Z inflating: build/bin/c10_Bitset_test 2023-03-08T02:50:44.2694959Z inflating: build/bin/c10_C++17_test 2023-03-08T02:50:44.2745841Z inflating: build/bin/c10_ConstexprCrc_test 2023-03-08T02:50:44.2798076Z inflating: build/bin/c10_DeadlockDetection_test 2023-03-08T02:50:44.2850472Z inflating: build/bin/c10_Half_test 2023-03-08T02:50:44.2910812Z inflating: build/bin/c10_LeftRight_test 2023-03-08T02:50:44.2976897Z inflating: build/bin/c10_Metaprogramming_test 2023-03-08T02:50:44.3131753Z inflating: build/bin/c10_SmallVectorTest 2023-03-08T02:50:44.3185224Z inflating: build/bin/c10_Synchronized_test 2023-03-08T02:50:44.3245431Z inflating: build/bin/c10_ThreadLocal_test 2023-03-08T02:50:44.3301474Z inflating: build/bin/c10_TypeIndex_test 2023-03-08T02:50:44.3355159Z inflating: build/bin/c10_TypeList_test 2023-03-08T02:50:44.3406126Z inflating: build/bin/c10_TypeTraits_test 2023-03-08T02:50:44.3461614Z inflating: build/bin/c10_accumulate_test 2023-03-08T02:50:44.3521186Z inflating: build/bin/c10_bfloat16_test 2023-03-08T02:50:44.3580036Z inflating: build/bin/c10_complex_math_test 2023-03-08T02:50:44.3638756Z inflating: build/bin/c10_complex_test 2023-03-08T02:50:44.3754232Z inflating: build/bin/c10_either_test 2023-03-08T02:50:44.3809791Z inflating: build/bin/c10_exception_test 2023-03-08T02:50:44.3862803Z inflating: build/bin/c10_flags_test 2023-03-08T02:50:44.4041390Z inflating: build/bin/c10_intrusive_ptr_test 2023-03-08T02:50:44.4094816Z inflating: build/bin/c10_irange_test 2023-03-08T02:50:44.4155359Z inflating: build/bin/c10_logging_test 2023-03-08T02:50:44.4233492Z inflating: build/bin/c10_optional_test 2023-03-08T02:50:44.4298976Z inflating: build/bin/c10_ordered_preserving_dict_test 2023-03-08T02:50:44.4357136Z inflating: build/bin/c10_registry_test 2023-03-08T02:50:44.4418782Z inflating: build/bin/c10_string_view_test 2023-03-08T02:50:44.4474025Z inflating: build/bin/c10_tempfile_test 2023-03-08T02:50:44.4532661Z inflating: build/bin/c10_typeid_test 2023-03-08T02:50:44.4591294Z inflating: build/bin/c10_intrusive_ptr_benchmark 2023-03-08T02:50:44.5093761Z inflating: build/bin/protoc-3.13.0.0 2023-03-08T02:50:44.5594813Z inflating: build/bin/protoc 2023-03-08T02:50:44.5651070Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2023-03-08T02:50:44.5708359Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2023-03-08T02:50:44.5764112Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2023-03-08T02:50:44.5819536Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2023-03-08T02:50:44.5876910Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2023-03-08T02:50:44.5932482Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2023-03-08T02:50:44.5989389Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2023-03-08T02:50:44.6040431Z inflating: build/bin/c10_cuda_CUDATest 2023-03-08T02:50:44.6355404Z inflating: build/bin/vec_test_all_types_DEFAULT 2023-03-08T02:50:44.6702473Z inflating: build/bin/vec_test_all_types_AVX2 2023-03-08T02:50:44.6759603Z inflating: build/bin/HashStoreTest 2023-03-08T02:50:44.6822726Z inflating: build/bin/TCPStoreTest 2023-03-08T02:50:44.6879963Z inflating: build/bin/FileStoreTest 2023-03-08T02:50:44.6895751Z inflating: build/bin/ProcessGroupMPITest 2023-03-08T02:50:44.6960523Z inflating: build/bin/test_edge_op_registration 2023-03-08T02:50:44.6963311Z inflating: build/bin/example_allreduce 2023-03-08T02:50:44.7020147Z inflating: build/bin/Dimname_test 2023-03-08T02:50:44.7096986Z inflating: build/bin/Dict_test 2023-03-08T02:50:44.7164908Z inflating: build/bin/MaybeOwned_test 2023-03-08T02:50:44.7225512Z inflating: build/bin/NamedTensor_test 2023-03-08T02:50:44.7288021Z inflating: build/bin/atest 2023-03-08T02:50:44.7350888Z inflating: build/bin/apply_utils_test 2023-03-08T02:50:44.7415713Z inflating: build/bin/basic 2023-03-08T02:50:44.7473393Z inflating: build/bin/broadcast_test 2023-03-08T02:50:44.7534816Z inflating: build/bin/cpu_generator_test 2023-03-08T02:50:44.7590591Z inflating: build/bin/cpu_profiling_allocator_test 2023-03-08T02:50:44.7643470Z inflating: build/bin/dispatch_key_set_test 2023-03-08T02:50:44.7737546Z inflating: build/bin/cpu_rng_test 2023-03-08T02:50:44.7790381Z inflating: build/bin/dlconvertor_test 2023-03-08T02:50:44.7852517Z inflating: build/bin/extension_backend_test 2023-03-08T02:50:44.7911364Z inflating: build/bin/half_test 2023-03-08T02:50:44.8011173Z inflating: build/bin/ivalue_test 2023-03-08T02:50:44.8063210Z inflating: build/bin/lazy_tensor_test 2023-03-08T02:50:44.8120802Z inflating: build/bin/math_kernel_test 2023-03-08T02:50:44.8177853Z inflating: build/bin/memory_format_test 2023-03-08T02:50:44.8235164Z inflating: build/bin/memory_overlapping_test 2023-03-08T02:50:44.8291117Z inflating: build/bin/mobile_memory_cleanup 2023-03-08T02:50:44.8345093Z inflating: build/bin/operator_name_test 2023-03-08T02:50:44.8404437Z inflating: build/bin/native_test 2023-03-08T02:50:44.8457621Z inflating: build/bin/operators_test 2023-03-08T02:50:44.8514489Z inflating: build/bin/packedtensoraccessor_test 2023-03-08T02:50:44.8583848Z inflating: build/bin/pow_test 2023-03-08T02:50:44.8644996Z inflating: build/bin/quantized_test 2023-03-08T02:50:44.8697181Z inflating: build/bin/reduce_ops_test 2023-03-08T02:50:44.8751574Z inflating: build/bin/reportMemoryUsage_test 2023-03-08T02:50:44.8807177Z inflating: build/bin/StorageUtils_test 2023-03-08T02:50:44.8867726Z inflating: build/bin/scalar_tensor_test 2023-03-08T02:50:44.8928400Z inflating: build/bin/scalar_test 2023-03-08T02:50:44.8984049Z inflating: build/bin/stride_properties_test 2023-03-08T02:50:44.9042276Z inflating: build/bin/type_ptr_test 2023-03-08T02:50:44.9125957Z inflating: build/bin/tensor_iterator_test 2023-03-08T02:50:44.9128577Z inflating: build/bin/thread_init_test 2023-03-08T02:50:44.9187610Z inflating: build/bin/test_parallel 2023-03-08T02:50:44.9251782Z inflating: build/bin/type_test 2023-03-08T02:50:44.9307909Z inflating: build/bin/undefined_tensor_test 2023-03-08T02:50:44.9360447Z inflating: build/bin/variant_test 2023-03-08T02:50:44.9361185Z inflating: build/bin/verify_api_visibility 2023-03-08T02:50:44.9434506Z inflating: build/bin/legacy_vmap_test 2023-03-08T02:50:44.9489344Z inflating: build/bin/weakref_test 2023-03-08T02:50:44.9543267Z inflating: build/bin/wrapdim_test 2023-03-08T02:50:44.9606703Z inflating: build/bin/IListRef_test 2023-03-08T02:50:44.9722882Z inflating: build/bin/List_test 2023-03-08T02:50:44.9774721Z inflating: build/bin/xla_tensor_test 2023-03-08T02:50:44.9903563Z inflating: build/bin/kernel_function_legacy_test 2023-03-08T02:50:45.0004422Z inflating: build/bin/kernel_function_test 2023-03-08T02:50:45.0073545Z inflating: build/bin/KernelFunction_test 2023-03-08T02:50:45.0209106Z inflating: build/bin/kernel_lambda_legacy_test 2023-03-08T02:50:45.0318136Z inflating: build/bin/kernel_lambda_test 2023-03-08T02:50:45.0381975Z inflating: build/bin/kernel_stackbased_test 2023-03-08T02:50:45.0436171Z inflating: build/bin/CppSignature_test 2023-03-08T02:50:45.0537210Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2023-03-08T02:50:45.0588106Z inflating: build/bin/op_allowlist_test 2023-03-08T02:50:45.0645682Z inflating: build/bin/inline_container_test 2023-03-08T02:50:45.0705785Z inflating: build/bin/backend_fallback_test 2023-03-08T02:50:45.1010459Z inflating: build/bin/op_registration_test 2023-03-08T02:50:45.1066190Z inflating: build/bin/cuda_apply_test 2023-03-08T02:50:45.1129524Z inflating: build/bin/cuda_atomic_ops_test 2023-03-08T02:50:45.1187728Z inflating: build/bin/cuda_caching_host_allocator_test 2023-03-08T02:50:45.1239761Z inflating: build/bin/cuda_device_test 2023-03-08T02:50:45.1314580Z inflating: build/bin/cuda_complex_math_test 2023-03-08T02:50:45.1377048Z inflating: build/bin/cuda_complex_test 2023-03-08T02:50:45.1439635Z inflating: build/bin/cuda_cub_test 2023-03-08T02:50:45.1492820Z inflating: build/bin/cuda_dlconvertor_test 2023-03-08T02:50:45.1546625Z inflating: build/bin/cuda_integer_divider_test 2023-03-08T02:50:45.1619503Z inflating: build/bin/cuda_distributions_test 2023-03-08T02:50:45.1682063Z inflating: build/bin/cuda_generator_test 2023-03-08T02:50:45.1733765Z inflating: build/bin/cuda_half_test 2023-03-08T02:50:45.1791456Z inflating: build/bin/cuda_reportMemoryUsage_test 2023-03-08T02:50:45.1856092Z inflating: build/bin/cuda_stream_test 2023-03-08T02:50:45.1907822Z inflating: build/bin/cuda_optional_test 2023-03-08T02:50:45.1962987Z inflating: build/bin/cuda_packedtensoraccessor_test 2023-03-08T02:50:45.2014556Z inflating: build/bin/cuda_cudnn_test 2023-03-08T02:50:45.2070494Z inflating: build/bin/cuda_vectorized_test 2023-03-08T02:50:45.2087208Z inflating: build/bin/tutorial_tensorexpr 2023-03-08T02:50:45.2157045Z inflating: build/bin/ProcessGroupGlooTest 2023-03-08T02:50:45.2219033Z inflating: build/bin/ProcessGroupGlooAsyncTest 2023-03-08T02:50:45.2284093Z inflating: build/bin/ProcessGroupNCCLTest 2023-03-08T02:50:45.2346028Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2023-03-08T02:50:45.2402805Z inflating: build/bin/ProcessGroupUCCTest 2023-03-08T02:50:45.2459788Z inflating: build/bin/test_dist_autograd 2023-03-08T02:50:45.2533858Z inflating: build/bin/test_cpp_rpc 2023-03-08T02:50:45.2535915Z inflating: build/bin/parallel_benchmark 2023-03-08T02:50:45.2608335Z inflating: build/bin/test_mobile_nnc 2023-03-08T02:50:45.2619140Z inflating: build/bin/aot_model_compiler_test 2023-03-08T02:50:45.2995531Z inflating: build/bin/test_lazy 2023-03-08T02:50:45.3871825Z inflating: build/bin/test_tensorexpr 2023-03-08T02:50:45.3877128Z inflating: build/bin/torch_shm_manager 2023-03-08T02:50:45.4486193Z inflating: build/bin/test_jit 2023-03-08T02:50:45.5766317Z inflating: build/bin/test_api 2023-03-08T02:50:45.6381834Z inflating: build/bin/nvfuser_tests 2023-03-08T02:50:45.6420132Z ##[group]Run df -H 2023-03-08T02:50:45.6420375Z df -H 2023-03-08T02:50:45.6439009Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:50:45.6439276Z env: 2023-03-08T02:50:45.6439514Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:45.6439764Z GPU_FLAG: --gpus all 2023-03-08T02:50:45.6439978Z ##[endgroup] 2023-03-08T02:50:45.6491803Z Filesystem Size Used Avail Use% Mounted on 2023-03-08T02:50:45.6492313Z /dev/root 1.1T 225G 842G 22% / 2023-03-08T02:50:45.6492812Z devtmpfs 45G 0 45G 0% /dev 2023-03-08T02:50:45.6493272Z tmpfs 45G 0 45G 0% /dev/shm 2023-03-08T02:50:45.6493663Z tmpfs 9.0G 1.1M 9.0G 1% /run 2023-03-08T02:50:45.6494541Z tmpfs 5.3M 0 5.3M 0% /run/lock 2023-03-08T02:50:45.6495083Z tmpfs 45G 0 45G 0% /sys/fs/cgroup 2023-03-08T02:50:45.6495673Z /dev/loop0 16M 16M 0 100% /snap/aws-cli/130 2023-03-08T02:50:45.6495968Z /dev/loop1 123M 123M 0 100% /snap/core/14399 2023-03-08T02:50:45.6496257Z /dev/loop2 123M 123M 0 100% /snap/core/14784 2023-03-08T02:50:45.6496542Z /dev/loop3 59M 59M 0 100% /snap/core18/2654 2023-03-08T02:50:45.6496833Z /dev/loop5 67M 67M 0 100% /snap/core20/1822 2023-03-08T02:50:45.6497103Z /dev/loop4 59M 59M 0 100% /snap/core18/2697 2023-03-08T02:50:45.6497444Z /dev/loop6 67M 67M 0 100% /snap/core20/1828 2023-03-08T02:50:45.6497723Z /dev/loop9 97M 97M 0 100% /snap/lxd/23991 2023-03-08T02:50:45.6498118Z /dev/loop8 355M 355M 0 100% /snap/google-cloud-sdk/322 2023-03-08T02:50:45.6498445Z /dev/loop10 97M 97M 0 100% /snap/lxd/24061 2023-03-08T02:50:45.6498729Z /dev/loop11 53M 53M 0 100% /snap/snapd/17883 2023-03-08T02:50:45.6498994Z /dev/loop12 53M 53M 0 100% /snap/snapd/18357 2023-03-08T02:50:45.6499270Z /dev/sda15 110M 5.5M 104M 5% /boot/efi 2023-03-08T02:50:45.6499666Z /dev/loop13 355M 355M 0 100% /snap/google-cloud-sdk/324 2023-03-08T02:50:45.6521683Z ##[group]Run .github/scripts/parse_ref.py 2023-03-08T02:50:45.6522001Z .github/scripts/parse_ref.py 2023-03-08T02:50:45.6538099Z shell: /usr/bin/bash -e {0} 2023-03-08T02:50:45.6538318Z env: 2023-03-08T02:50:45.6538542Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:45.6538792Z GPU_FLAG: --gpus all 2023-03-08T02:50:45.6539006Z ##[endgroup] 2023-03-08T02:50:45.6831842Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-03-08T02:50:45.6832203Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-03-08T02:50:45.6848773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T02:50:45.6849050Z env: 2023-03-08T02:50:45.6849258Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:45.6849501Z GPU_FLAG: --gpus all 2023-03-08T02:50:45.6849727Z JOB_TIMEOUT: 1200 2023-03-08T02:50:45.6849929Z ##[endgroup] 2023-03-08T02:50:45.6935751Z ##[group]Run set -x 2023-03-08T02:50:45.6936097Z set -x 2023-03-08T02:50:45.6936308Z  2023-03-08T02:50:45.6936544Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2023-03-08T02:50:45.6936861Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2023-03-08T02:50:45.6937179Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2023-03-08T02:50:45.6937456Z  TEST_COMMAND=.ci/onnx/test.sh 2023-03-08T02:50:45.6937704Z else 2023-03-08T02:50:45.6937953Z  TEST_COMMAND=.ci/pytorch/test.sh 2023-03-08T02:50:45.6938184Z fi 2023-03-08T02:50:45.6938387Z  2023-03-08T02:50:45.6938688Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2023-03-08T02:50:45.6938968Z  2023-03-08T02:50:45.6939241Z # sanitize the input commit message and PR body here: 2023-03-08T02:50:45.6939511Z # 2023-03-08T02:50:45.6939846Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2023-03-08T02:50:45.6940441Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2023-03-08T02:50:45.6940833Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2023-03-08T02:50:45.6941125Z PR_BODY="${PR_BODY//[$'\n\r']}" 2023-03-08T02:50:45.6941344Z  2023-03-08T02:50:45.6941677Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2023-03-08T02:50:45.6942028Z # wreak havoc internally 2023-03-08T02:50:45.6942312Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2023-03-08T02:50:45.6942613Z export PR_BODY="${PR_BODY//[\'\"]}" 2023-03-08T02:50:45.6942916Z  2023-03-08T02:50:45.6943205Z # detached container should get cleaned up by teardown_ec2_linux 2023-03-08T02:50:45.6943562Z # TODO: Stop building test binaries as part of the build phase 2023-03-08T02:50:45.6943911Z # Used for GPU_FLAG since that doesn't play nice 2023-03-08T02:50:45.6944219Z # shellcheck disable=SC2086,SC2090 2023-03-08T02:50:45.6944486Z container_name=$(docker run \ 2023-03-08T02:50:45.6944768Z  ${GPU_FLAG:-} \ 2023-03-08T02:50:45.6945019Z  -e BUILD_ENVIRONMENT \ 2023-03-08T02:50:45.6945257Z  -e PR_NUMBER \ 2023-03-08T02:50:45.6945502Z  -e GITHUB_ACTIONS \ 2023-03-08T02:50:45.6945747Z  -e BASE_SHA \ 2023-03-08T02:50:45.6945977Z  -e BRANCH \ 2023-03-08T02:50:45.6946185Z  -e SHA1 \ 2023-03-08T02:50:45.6946425Z  -e AWS_DEFAULT_REGION \ 2023-03-08T02:50:45.6946678Z  -e IN_WHEEL_TEST \ 2023-03-08T02:50:45.6946906Z  -e SHARD_NUMBER \ 2023-03-08T02:50:45.6947342Z  -e TEST_CONFIG \ 2023-03-08T02:50:45.6947592Z  -e NUM_TEST_SHARDS \ 2023-03-08T02:50:45.6947816Z  -e PR_BODY \ 2023-03-08T02:50:45.6948060Z  -e COMMIT_MESSAGES \ 2023-03-08T02:50:45.6948328Z  -e CONTINUE_THROUGH_ERROR \ 2023-03-08T02:50:45.6948592Z  -e PYTORCH_RETRY_TEST_CASES \ 2023-03-08T02:50:45.6948881Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2023-03-08T02:50:45.6949148Z  -e PR_LABELS \ 2023-03-08T02:50:45.6949400Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2023-03-08T02:50:45.6949672Z  -e SCCACHE_BUCKET \ 2023-03-08T02:50:45.6949926Z  -e SCCACHE_S3_KEY_PREFIX \ 2023-03-08T02:50:45.6950172Z  -e XLA_CUDA \ 2023-03-08T02:50:45.6950419Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2023-03-08T02:50:45.6950715Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2023-03-08T02:50:45.6951021Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2023-03-08T02:50:45.6951311Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2023-03-08T02:50:45.6951630Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2023-03-08T02:50:45.6952043Z  --ulimit stack=10485760:83886080 \ 2023-03-08T02:50:45.6952329Z  --security-opt seccomp=unconfined \ 2023-03-08T02:50:45.6952610Z  --cap-add=SYS_PTRACE \ 2023-03-08T02:50:45.6952903Z  --ipc=host \ 2023-03-08T02:50:45.6953149Z  --shm-size="${SHM_SIZE}" \ 2023-03-08T02:50:45.6953375Z  --tty \ 2023-03-08T02:50:45.6953594Z  --detach \ 2023-03-08T02:50:45.6953842Z  --name="${container_name}" \ 2023-03-08T02:50:45.6954080Z  --user jenkins \ 2023-03-08T02:50:45.6954378Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2023-03-08T02:50:45.6954711Z  -w /var/lib/jenkins/workspace \ 2023-03-08T02:50:45.6954959Z  "${DOCKER_IMAGE}" 2023-03-08T02:50:45.6955183Z ) 2023-03-08T02:50:45.6955460Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2023-03-08T02:50:45.6955865Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2023-03-08T02:50:45.6971943Z shell: /usr/bin/bash -e {0} 2023-03-08T02:50:45.6972173Z env: 2023-03-08T02:50:45.6972397Z GIT_DEFAULT_BRANCH: master 2023-03-08T02:50:45.6972630Z GPU_FLAG: --gpus all 2023-03-08T02:50:45.6972997Z BUILD_ENVIRONMENT: linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:50:45.6973296Z PR_NUMBER: 2023-03-08T02:50:45.6973499Z BRANCH: master 2023-03-08T02:50:45.6973763Z SHA1: c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:50:45.6974072Z BASE_SHA: c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:50:45.6974338Z PYTORCH_RETRY_TEST_CASES: 1 2023-03-08T02:50:45.6974602Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2023-03-08T02:50:45.6974886Z TEST_CONFIG: inductor_huggingface_perf 2023-03-08T02:50:45.6975125Z SHARD_NUMBER: 1 2023-03-08T02:50:45.6975352Z NUM_TEST_SHARDS: 1 2023-03-08T02:50:45.6975573Z PR_BODY: 2023-03-08T02:50:45.6975790Z CONTINUE_THROUGH_ERROR: False 2023-03-08T02:50:45.6976104Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2023-03-08T02:50:45.6976440Z SCCACHE_S3_KEY_PREFIX: inductor-A100-perf 2023-03-08T02:50:45.6976705Z SHM_SIZE: 2g 2023-03-08T02:50:45.6977158Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:50:45.6977612Z XLA_CUDA: 2023-03-08T02:50:45.6977937Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2023-03-08T02:50:45.6978279Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2023-03-08T02:50:45.6978557Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2023-03-08T02:50:45.6978804Z ##[endgroup] 2023-03-08T02:50:45.7012812Z + [[ inductor_huggingface_perf == \m\u\l\t\i\g\p\u ]] 2023-03-08T02:50:45.7013584Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *onnx* ]] 2023-03-08T02:50:45.7013904Z + TEST_COMMAND=.ci/pytorch/test.sh 2023-03-08T02:50:45.7016627Z ++ git cherry -v origin/master 2023-03-08T02:50:45.7035301Z + COMMIT_MESSAGES= 2023-03-08T02:50:45.7035957Z + COMMIT_MESSAGES= 2023-03-08T02:50:45.7036308Z + PR_BODY= 2023-03-08T02:50:45.7036575Z + export COMMIT_MESSAGES= 2023-03-08T02:50:45.7036937Z + COMMIT_MESSAGES= 2023-03-08T02:50:45.7037223Z + export PR_BODY= 2023-03-08T02:50:45.7037438Z + PR_BODY= 2023-03-08T02:50:45.7047581Z +++ nproc --ignore=2 2023-03-08T02:50:45.7059694Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=10 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 --env-file=/tmp/github_env_4360459833 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/weiwangmeta/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T02:50:50.4078163Z + container_name=e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T02:50:50.4078647Z + echo DOCKER_CONTAINER_ID=e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T02:50:50.4079806Z ++ echo dist/torch-2.1.0a0+gitc88aa33-cp310-cp310-linux_x86_64.whl 2023-03-08T02:50:50.4082371Z + docker exec -t e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 sh -c 'pip install dist/torch-2.1.0a0+gitc88aa33-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2023-03-08T02:50:50.9398653Z Processing ./dist/torch-2.1.0a0+gitc88aa33-cp310-cp310-linux_x86_64.whl 2023-03-08T02:50:51.8191878Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (3.1.2) 2023-03-08T02:50:51.8196016Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (2.6.3) 2023-03-08T02:50:51.8200088Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (4.5.0) 2023-03-08T02:50:51.8204381Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (1.11.1) 2023-03-08T02:50:51.8208254Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (3.9.0) 2023-03-08T02:50:51.8221361Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+gitc88aa33) (3.3.0) 2023-03-08T02:50:51.8288808Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.1.0a0+gitc88aa33) (1.21.2) 2023-03-08T02:50:51.8382376Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.1.0a0+gitc88aa33) (2.1.2) 2023-03-08T02:50:51.8560550Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.1.0a0+gitc88aa33) (1.3.0) 2023-03-08T02:50:52.6722913Z Installing collected packages: torch 2023-03-08T02:50:52.6723478Z Attempting uninstall: torch 2023-03-08T02:50:52.6736236Z Found existing installation: torch 1.13.1 2023-03-08T02:50:53.7354816Z Uninstalling torch-1.13.1: 2023-03-08T02:50:56.4470632Z Successfully uninstalled torch-1.13.1 2023-03-08T02:51:05.7990183Z Successfully installed torch-2.1.0a0+gitc88aa33 2023-03-08T02:51:05.8709691Z + echo 'Environment variables:' 2023-03-08T02:51:05.8710176Z Environment variables: 2023-03-08T02:51:05.8710459Z + env 2023-03-08T02:51:05.8719907Z SHARD_NUMBER=1 2023-03-08T02:51:05.8720661Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-03-08T02:51:05.8721280Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-03-08T02:51:05.8721830Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-03-08T02:51:05.8722509Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.15.5-1+cuda11.8 2023-03-08T02:51:05.8722863Z UCC_HOME=/usr 2023-03-08T02:51:05.8723266Z BUILD_ENVIRONMENT=linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:51:05.8723596Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-03-08T02:51:05.8723968Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-03-08T02:51:05.8724221Z INSTALLED_DB=yes 2023-03-08T02:51:05.8724447Z HOSTNAME=e5067bc61540 2023-03-08T02:51:05.8724735Z GITHUB_REF_NAME=master 2023-03-08T02:51:05.8725092Z GITHUB_API_URL=https://api.github.com 2023-03-08T02:51:05.8725382Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-03-08T02:51:05.8725637Z OPENSSL_DIR=/opt/openssl 2023-03-08T02:51:05.8725904Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee 2023-03-08T02:51:05.8726838Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8727253Z CUDA_PATH=/usr/local/cuda 2023-03-08T02:51:05.8727737Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-03-08T02:51:05.8728156Z GITHUB_RUN_ATTEMPT=1 2023-03-08T02:51:05.8728416Z TEST_CONFIG=inductor_huggingface_perf 2023-03-08T02:51:05.8728701Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-03-08T02:51:05.8729061Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-03-08T02:51:05.8729353Z GITHUB_REPOSITORY_OWNER=pytorch 2023-03-08T02:51:05.8729594Z GITHUB_ACTIONS=true 2023-03-08T02:51:05.8729827Z NVIDIA_VISIBLE_DEVICES=all 2023-03-08T02:51:05.8730104Z NV_NVPROF_VERSION=11.8.87-1 2023-03-08T02:51:05.8730391Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-03-08T02:51:05.8730888Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/master 2023-03-08T02:51:05.8731258Z NVIDIA_PRODUCT_NAME=CUDA 2023-03-08T02:51:05.8731596Z CI=true 2023-03-08T02:51:05.8731810Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-03-08T02:51:05.8732200Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-03-08T02:51:05.8732476Z BRANCH=master 2023-03-08T02:51:05.8732682Z GITHUB_HEAD_REF= 2023-03-08T02:51:05.8733111Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab 2023-03-08T02:51:05.8733415Z GITHUB_ACTOR=pytorchmergebot 2023-03-08T02:51:05.8733709Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-03-08T02:51:05.8733964Z GITHUB_ACTION_REF= 2023-03-08T02:51:05.8734218Z NCCL_VERSION=2.15.5-1 2023-03-08T02:51:05.8734452Z GITHUB_ACTION=__self 2023-03-08T02:51:05.8734683Z GITHUB_REF_PROTECTED=true 2023-03-08T02:51:05.8735119Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-03-08T02:51:05.8735472Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-03-08T02:51:05.8737481Z *** 2023-03-08T02:51:05.8737726Z INSTALLED_VISION=yes 2023-03-08T02:51:05.8738002Z NVARCH=x86_64 2023-03-08T02:51:05.8738334Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-03-08T02:51:05.8738599Z HOME=/var/lib/jenkins 2023-03-08T02:51:05.8739137Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8739530Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-03-08T02:51:05.8739777Z GITHUB_ACTION_REPOSITORY= 2023-03-08T02:51:05.8740020Z GITHUB_REF_TYPE=branch 2023-03-08T02:51:05.8740306Z NV_LIBNCCL_PACKAGE_VERSION=2.15.5-1 2023-03-08T02:51:05.8740549Z GITHUB_RETENTION_DAYS=90 2023-03-08T02:51:05.8740916Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-03-08T02:51:05.8741306Z NV_LIBNCCL_PACKAGE=libnccl2=2.15.5-1+cuda11.8 2023-03-08T02:51:05.8741978Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8742364Z DEBIAN_FRONTEND=noninteractive 2023-03-08T02:51:05.8742746Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-03-08T02:51:05.8743019Z GITHUB_REF=refs/heads/master 2023-03-08T02:51:05.8743310Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-03-08T02:51:05.8743599Z GITHUB_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.8743878Z INSTALLED_PROTOBUF=yes 2023-03-08T02:51:05.8744115Z ANACONDA_PYTHON_VERSION=3.10 2023-03-08T02:51:05.8744366Z GITHUB_REPOSITORY_ID=65600975 2023-03-08T02:51:05.8744607Z GITHUB_RUN_ID=4360459833 2023-03-08T02:51:05.8744918Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-03-08T02:51:05.8745191Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-03-08T02:51:05.8745462Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-03-08T02:51:05.8745736Z NV_NVTX_VERSION=11.8.86-1 2023-03-08T02:51:05.8745983Z CONTINUE_THROUGH_ERROR=False 2023-03-08T02:51:05.8746259Z GITHUB_SERVER_URL=https://github.com 2023-03-08T02:51:05.8746495Z MAX_JOBS=10 2023-03-08T02:51:05.8746714Z GITHUB_ACTOR_ID=97764156 2023-03-08T02:51:05.8746991Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-03-08T02:51:05.8747627Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-03-08T02:51:05.8748353Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json 2023-03-08T02:51:05.8748689Z UCX_HOME=/usr 2023-03-08T02:51:05.8748911Z PYTORCH_RETRY_TEST_CASES=1 2023-03-08T02:51:05.8749217Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-03-08T02:51:05.8749549Z BASE_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.8749860Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-03-08T02:51:05.8750100Z PR_BODY= 2023-03-08T02:51:05.8750307Z GITHUB_BASE_REF= 2023-03-08T02:51:05.8750509Z TERM=xterm 2023-03-08T02:51:05.8750707Z XLA_CUDA= 2023-03-08T02:51:05.8750957Z NV_NVML_DEV_VERSION=11.8.86-1 2023-03-08T02:51:05.8751194Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-03-08T02:51:05.8751431Z CUDA_VERSION=11.8.0 2023-03-08T02:51:05.8751753Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-03-08T02:51:05.8752010Z OPENSSL_ROOT_DIR=/opt/openssl 2023-03-08T02:51:05.8752557Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8752921Z GITHUB_JOB=test 2023-03-08T02:51:05.8753334Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf 2023-03-08T02:51:05.8753585Z COMMIT_MESSAGES= 2023-03-08T02:51:05.8753851Z NVIDIA_DRIVER_CAPABILITIES=compute,utility 2023-03-08T02:51:05.8754120Z NUM_TEST_SHARDS=1 2023-03-08T02:51:05.8754320Z PR_NUMBER= 2023-03-08T02:51:05.8754849Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8755201Z SHLVL=1 2023-03-08T02:51:05.8755515Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-03-08T02:51:05.8755820Z GITHUB_REPOSITORY=pytorch/pytorch 2023-03-08T02:51:05.8757348Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-03-08T02:51:05.8759011Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-03-08T02:51:05.8759288Z SHA1=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.8759569Z GITHUB_EVENT_NAME=schedule 2023-03-08T02:51:05.8759855Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-03-08T02:51:05.8760174Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-03-08T02:51:05.8760438Z GITHUB_RUN_NUMBER=420 2023-03-08T02:51:05.8760757Z GITHUB_WORKFLOW=inductor-A100-perf 2023-03-08T02:51:05.8761185Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-03-08T02:51:05.8761633Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.15.5-1 2023-03-08T02:51:05.8761961Z GITHUB_WORKFLOW_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.8762409Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T02:51:05.8762758Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2023-03-08T02:51:05.8763043Z SKIP_SCCACHE_INITIALIZATION=1 2023-03-08T02:51:05.8763272Z _=/usr/bin/env 2023-03-08T02:51:05.8763641Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2023-03-08T02:51:05.8924594Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2023-03-08T02:51:05.8925487Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2023-03-08T02:51:05.8926109Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2023-03-08T02:51:05.8926595Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2023-03-08T02:51:05.8926904Z + BUILD_DIR=build 2023-03-08T02:51:05.8927153Z + BUILD_RENAMED_DIR=build_renamed 2023-03-08T02:51:05.8927474Z + BUILD_BIN_DIR=build/bin 2023-03-08T02:51:05.8927715Z + export VALGRIND=ON 2023-03-08T02:51:05.8927946Z + VALGRIND=ON 2023-03-08T02:51:05.8928315Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *clang9* ]] 2023-03-08T02:51:05.8928767Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 != *bazel* ]] 2023-03-08T02:51:05.8930346Z ++ realpath build/custom_test_artifacts 2023-03-08T02:51:05.8939899Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2023-03-08T02:51:05.8943590Z ++ dirname .ci/pytorch/test.sh 2023-03-08T02:51:05.8951932Z + source .ci/pytorch/common.sh 2023-03-08T02:51:05.8957160Z +++ dirname .ci/pytorch/common.sh 2023-03-08T02:51:05.8969698Z ++ source .ci/pytorch/common_utils.sh 2023-03-08T02:51:05.8971715Z +++ declare -f -t trap_add 2023-03-08T02:51:05.8979445Z ++ set -ex 2023-03-08T02:51:05.8980088Z ++ [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *rocm* ]] 2023-03-08T02:51:05.8980547Z ++ BUILD_TEST_LIBTORCH=0 2023-03-08T02:51:05.8980927Z + echo 'Environment variables' 2023-03-08T02:51:05.8981189Z Environment variables 2023-03-08T02:51:05.8981393Z + env 2023-03-08T02:51:05.8989088Z SHARD_NUMBER=1 2023-03-08T02:51:05.8989607Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-03-08T02:51:05.8990227Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-03-08T02:51:05.8990882Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-03-08T02:51:05.8991526Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.15.5-1+cuda11.8 2023-03-08T02:51:05.8991925Z UCC_HOME=/usr 2023-03-08T02:51:05.8992317Z BUILD_ENVIRONMENT=linux-bionic-cuda11.8-py3.10-gcc7-sm80 2023-03-08T02:51:05.8992832Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-03-08T02:51:05.8993452Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-03-08T02:51:05.8993860Z INSTALLED_DB=yes 2023-03-08T02:51:05.8994215Z HOSTNAME=e5067bc61540 2023-03-08T02:51:05.8994607Z GITHUB_REF_NAME=master 2023-03-08T02:51:05.8995077Z GITHUB_API_URL=https://api.github.com 2023-03-08T02:51:05.8995582Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-03-08T02:51:05.8996022Z OPENSSL_DIR=/opt/openssl 2023-03-08T02:51:05.8996496Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee 2023-03-08T02:51:05.8997684Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.8998382Z CUDA_PATH=/usr/local/cuda 2023-03-08T02:51:05.8999276Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-03-08T02:51:05.8999915Z GITHUB_RUN_ATTEMPT=1 2023-03-08T02:51:05.9000359Z TEST_CONFIG=inductor_huggingface_perf 2023-03-08T02:51:05.9000976Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-03-08T02:51:05.9001594Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-03-08T02:51:05.9002101Z GITHUB_REPOSITORY_OWNER=pytorch 2023-03-08T02:51:05.9002562Z GITHUB_ACTIONS=true 2023-03-08T02:51:05.9002944Z NVIDIA_VISIBLE_DEVICES=all 2023-03-08T02:51:05.9003412Z NV_NVPROF_VERSION=11.8.87-1 2023-03-08T02:51:05.9003868Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-03-08T02:51:05.9004684Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/master 2023-03-08T02:51:05.9005268Z NVIDIA_PRODUCT_NAME=CUDA 2023-03-08T02:51:05.9005620Z CI=true 2023-03-08T02:51:05.9005932Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-03-08T02:51:05.9006337Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-03-08T02:51:05.9006618Z BRANCH=master 2023-03-08T02:51:05.9006822Z GITHUB_HEAD_REF= 2023-03-08T02:51:05.9007308Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab 2023-03-08T02:51:05.9007696Z GITHUB_ACTOR=pytorchmergebot 2023-03-08T02:51:05.9007983Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-03-08T02:51:05.9008258Z GITHUB_ACTION_REF= 2023-03-08T02:51:05.9008522Z NCCL_VERSION=2.15.5-1 2023-03-08T02:51:05.9008744Z GITHUB_ACTION=__self 2023-03-08T02:51:05.9008966Z VALGRIND=ON 2023-03-08T02:51:05.9009196Z GITHUB_REF_PROTECTED=true 2023-03-08T02:51:05.9009619Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-03-08T02:51:05.9009978Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-03-08T02:51:05.9010356Z *** 2023-03-08T02:51:05.9010624Z INSTALLED_VISION=yes 2023-03-08T02:51:05.9010848Z NVARCH=x86_64 2023-03-08T02:51:05.9011114Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-03-08T02:51:05.9011370Z HOME=/var/lib/jenkins 2023-03-08T02:51:05.9011908Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.9012288Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-03-08T02:51:05.9012552Z GITHUB_ACTION_REPOSITORY= 2023-03-08T02:51:05.9012891Z GITHUB_REF_TYPE=branch 2023-03-08T02:51:05.9013169Z NV_LIBNCCL_PACKAGE_VERSION=2.15.5-1 2023-03-08T02:51:05.9013428Z GITHUB_RETENTION_DAYS=90 2023-03-08T02:51:05.9013802Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-03-08T02:51:05.9014182Z NV_LIBNCCL_PACKAGE=libnccl2=2.15.5-1+cuda11.8 2023-03-08T02:51:05.9014735Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.9015119Z DEBIAN_FRONTEND=noninteractive 2023-03-08T02:51:05.9015453Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-03-08T02:51:05.9015713Z GITHUB_REF=refs/heads/master 2023-03-08T02:51:05.9015994Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-03-08T02:51:05.9016284Z GITHUB_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.9016555Z INSTALLED_PROTOBUF=yes 2023-03-08T02:51:05.9016805Z ANACONDA_PYTHON_VERSION=3.10 2023-03-08T02:51:05.9017065Z GITHUB_REPOSITORY_ID=65600975 2023-03-08T02:51:05.9017294Z GITHUB_RUN_ID=4360459833 2023-03-08T02:51:05.9017686Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-03-08T02:51:05.9017968Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-03-08T02:51:05.9018229Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-03-08T02:51:05.9018514Z NV_NVTX_VERSION=11.8.86-1 2023-03-08T02:51:05.9018764Z CONTINUE_THROUGH_ERROR=False 2023-03-08T02:51:05.9019031Z GITHUB_SERVER_URL=https://github.com 2023-03-08T02:51:05.9019287Z MAX_JOBS=10 2023-03-08T02:51:05.9019509Z GITHUB_ACTOR_ID=97764156 2023-03-08T02:51:05.9019773Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-03-08T02:51:05.9020131Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-03-08T02:51:05.9020613Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json 2023-03-08T02:51:05.9020919Z UCX_HOME=/usr 2023-03-08T02:51:05.9021149Z PYTORCH_RETRY_TEST_CASES=1 2023-03-08T02:51:05.9021453Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-03-08T02:51:05.9021775Z BASE_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.9022101Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-03-08T02:51:05.9022342Z PR_BODY= 2023-03-08T02:51:05.9022553Z GITHUB_BASE_REF= 2023-03-08T02:51:05.9022754Z TERM=xterm 2023-03-08T02:51:05.9022954Z XLA_CUDA= 2023-03-08T02:51:05.9023202Z NV_NVML_DEV_VERSION=11.8.86-1 2023-03-08T02:51:05.9023439Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-03-08T02:51:05.9023677Z CUDA_VERSION=11.8.0 2023-03-08T02:51:05.9023999Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-03-08T02:51:05.9024259Z OPENSSL_ROOT_DIR=/opt/openssl 2023-03-08T02:51:05.9024803Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.9025167Z GITHUB_JOB=test 2023-03-08T02:51:05.9025479Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf 2023-03-08T02:51:05.9025744Z COMMIT_MESSAGES= 2023-03-08T02:51:05.9026009Z NVIDIA_DRIVER_CAPABILITIES=compute,utility 2023-03-08T02:51:05.9026256Z NUM_TEST_SHARDS=1 2023-03-08T02:51:05.9026532Z PR_NUMBER= 2023-03-08T02:51:05.9027065Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_9ade832d-c516-4b30-b862-5e7fa9510625 2023-03-08T02:51:05.9027766Z SHLVL=1 2023-03-08T02:51:05.9028117Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-03-08T02:51:05.9028423Z GITHUB_REPOSITORY=pytorch/pytorch 2023-03-08T02:51:05.9029941Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-03-08T02:51:05.9031631Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-03-08T02:51:05.9031927Z SHA1=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.9032204Z GITHUB_EVENT_NAME=schedule 2023-03-08T02:51:05.9032475Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-03-08T02:51:05.9032816Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-03-08T02:51:05.9033082Z GITHUB_RUN_NUMBER=420 2023-03-08T02:51:05.9033391Z GITHUB_WORKFLOW=inductor-A100-perf 2023-03-08T02:51:05.9033834Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-03-08T02:51:05.9034286Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.15.5-1 2023-03-08T02:51:05.9034613Z GITHUB_WORKFLOW_SHA=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e 2023-03-08T02:51:05.9035058Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T02:51:05.9035403Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2023-03-08T02:51:05.9035686Z SKIP_SCCACHE_INITIALIZATION=1 2023-03-08T02:51:05.9035916Z _=/usr/bin/env 2023-03-08T02:51:05.9036189Z + echo 'Testing pytorch' 2023-03-08T02:51:05.9036421Z Testing pytorch 2023-03-08T02:51:05.9036655Z + export LANG=C.UTF-8 2023-03-08T02:51:05.9036898Z + LANG=C.UTF-8 2023-03-08T02:51:05.9037110Z + PR_NUMBER= 2023-03-08T02:51:05.9037409Z + [[ inductor_huggingface_perf == \d\e\f\a\u\l\t ]] 2023-03-08T02:51:05.9037732Z + [[ inductor_huggingface_perf == \d\i\s\t\r\i\b\u\t\e\d ]] 2023-03-08T02:51:05.9038046Z + [[ inductor_huggingface_perf == \s\l\o\w ]] 2023-03-08T02:51:05.9038481Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *slow-gradcheck* ]] 2023-03-08T02:51:05.9038944Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *cuda* ]] 2023-03-08T02:51:05.9039280Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-03-08T02:51:05.9039560Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-03-08T02:51:05.9039856Z + [[ inductor_huggingface_perf == *crossref* ]] 2023-03-08T02:51:05.9040268Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *rocm* ]] 2023-03-08T02:51:05.9040714Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 != *-bazel-* ]] 2023-03-08T02:51:05.9041073Z + pip_install --user ninja==1.10.2 2023-03-08T02:51:05.9041448Z + pip install --progress-bar off --user ninja==1.10.2 2023-03-08T02:51:06.4289002Z Collecting ninja==1.10.2 2023-03-08T02:51:06.4908793Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2023-03-08T02:51:07.2730193Z Installing collected packages: ninja 2023-03-08T02:51:07.2820205Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2023-03-08T02:51:07.2820955Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-03-08T02:51:07.2877173Z Successfully installed ninja-1.10.2 2023-03-08T02:51:07.3598838Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-03-08T02:51:07.3599532Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-03-08T02:51:07.3600299Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *asan* ]] 2023-03-08T02:51:07.3600744Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *-tsan* ]] 2023-03-08T02:51:07.3601118Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2023-03-08T02:51:07.3601864Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2023-03-08T02:51:07.3602559Z + DYNAMO_BENCHMARK_FLAGS=() 2023-03-08T02:51:07.3602911Z + [[ inductor_huggingface_perf == *aot_eager* ]] 2023-03-08T02:51:07.3603224Z + [[ inductor_huggingface_perf == *inductor* ]] 2023-03-08T02:51:07.3603611Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2023-03-08T02:51:07.3603892Z + [[ inductor_huggingface_perf == *dynamic* ]] 2023-03-08T02:51:07.3604203Z + [[ inductor_huggingface_perf == *cpu_accuracy* ]] 2023-03-08T02:51:07.3604562Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2023-03-08T02:51:07.3610068Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *tbb* ]] 2023-03-08T02:51:07.3624493Z + [[ inductor_huggingface_perf == *_perf* ]] 2023-03-08T02:51:07.3624789Z + install_matplotlib 2023-03-08T02:51:07.3625032Z + pip_install matplotlib 2023-03-08T02:51:07.3625391Z + pip install --progress-bar off matplotlib 2023-03-08T02:51:08.0317821Z Collecting matplotlib 2023-03-08T02:51:08.1306260Z Downloading matplotlib-3.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB) 2023-03-08T02:51:08.3292339Z Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from matplotlib) (2.8.2) 2023-03-08T02:51:08.3298462Z Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from matplotlib) (23.0) 2023-03-08T02:51:08.4242102Z Collecting contourpy>=1.0.1 2023-03-08T02:51:08.4410845Z Downloading contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB) 2023-03-08T02:51:08.4491935Z Requirement already satisfied: pillow>=6.2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from matplotlib) (9.4.0) 2023-03-08T02:51:08.4971716Z Collecting pyparsing>=2.3.1 2023-03-08T02:51:08.5119600Z Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2023-03-08T02:51:08.5932369Z Collecting fonttools>=4.22.0 2023-03-08T02:51:08.6115882Z Downloading fonttools-4.39.0-py3-none-any.whl (1.0 MB) 2023-03-08T02:51:08.7000291Z Collecting kiwisolver>=1.0.1 2023-03-08T02:51:08.7174025Z Downloading kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2023-03-08T02:51:08.7384529Z Requirement already satisfied: numpy>=1.20 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from matplotlib) (1.21.2) 2023-03-08T02:51:08.7662045Z Collecting cycler>=0.10 2023-03-08T02:51:08.7803974Z Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB) 2023-03-08T02:51:08.8524399Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0) 2023-03-08T02:51:09.6471643Z Installing collected packages: pyparsing, kiwisolver, fonttools, cycler, contourpy, matplotlib 2023-03-08T02:51:11.7726172Z Successfully installed contourpy-1.0.7 cycler-0.11.0 fonttools-4.39.0 kiwisolver-1.4.4 matplotlib-3.7.1 pyparsing-3.0.9 2023-03-08T02:51:11.8689440Z + install_tabulate 2023-03-08T02:51:11.8690125Z + pip_install tabulate 2023-03-08T02:51:11.8690703Z + pip install --progress-bar off tabulate 2023-03-08T02:51:12.3531673Z Collecting tabulate 2023-03-08T02:51:12.4132079Z Downloading tabulate-0.9.0-py3-none-any.whl (35 kB) 2023-03-08T02:51:13.2247552Z Installing collected packages: tabulate 2023-03-08T02:51:13.2692552Z Successfully installed tabulate-0.9.0 2023-03-08T02:51:13.3408078Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *libtorch* ]] 2023-03-08T02:51:13.3408895Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *-bazel-* ]] 2023-03-08T02:51:13.3409556Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *-tsan* ]] 2023-03-08T02:51:13.3410092Z + cd test 2023-03-08T02:51:13.3412736Z + python -c 'import torch; print(torch.__config__.show())' 2023-03-08T02:51:14.8192972Z PyTorch built with: 2023-03-08T02:51:14.8193851Z - GCC 7.5 2023-03-08T02:51:14.8194126Z - C++ Version: 201703 2023-03-08T02:51:14.8194693Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-03-08T02:51:14.8195601Z - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e) 2023-03-08T02:51:14.8196014Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-03-08T02:51:14.8196370Z - LAPACK is enabled (usually provided by MKL) 2023-03-08T02:51:14.8196665Z - NNPACK is enabled 2023-03-08T02:51:14.8196957Z - CPU capability usage: AVX2 2023-03-08T02:51:14.8197247Z - CUDA Runtime 11.8 2023-03-08T02:51:14.8197616Z - NVCC architecture flags: -gencode;arch=compute_80,code=sm_80 2023-03-08T02:51:14.8198481Z - CuDNN 8.7 2023-03-08T02:51:14.8198833Z - Magma 2.6.1 2023-03-08T02:51:14.8202230Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2023-03-08T02:51:14.8204385Z 2023-03-08T02:51:15.1114990Z + cd test 2023-03-08T02:51:15.1115684Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2023-03-08T02:51:16.4818205Z ATen/Parallel: 2023-03-08T02:51:16.4834507Z at::get_num_threads() : 6 2023-03-08T02:51:16.4834842Z at::get_num_interop_threads() : 6 2023-03-08T02:51:16.4835098Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-03-08T02:51:16.4835352Z omp_get_max_threads() : 6 2023-03-08T02:51:16.4836058Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-03-08T02:51:16.4836409Z mkl_get_max_threads() : 6 2023-03-08T02:51:16.4836851Z Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e) 2023-03-08T02:51:16.4837210Z std::thread::hardware_concurrency() : 12 2023-03-08T02:51:16.4837460Z Environment variables: 2023-03-08T02:51:16.4837704Z OMP_NUM_THREADS : [not set] 2023-03-08T02:51:16.4837947Z MKL_NUM_THREADS : [not set] 2023-03-08T02:51:16.4838183Z ATen parallel backend: OpenMP 2023-03-08T02:51:16.4838358Z 2023-03-08T02:51:16.7466099Z + [[ inductor_huggingface_perf == *backward* ]] 2023-03-08T02:51:16.7466755Z + [[ inductor_huggingface_perf == *xla* ]] 2023-03-08T02:51:16.7467324Z + [[ inductor_huggingface_perf == \j\i\t\_\l\e\g\a\c\y ]] 2023-03-08T02:51:16.7468364Z + [[ linux-bionic-cuda11.8-py3.10-gcc7-sm80 == *libtorch* ]] 2023-03-08T02:51:16.7468895Z + [[ inductor_huggingface_perf == distributed ]] 2023-03-08T02:51:16.7469417Z + [[ inductor_huggingface_perf == deploy ]] 2023-03-08T02:51:16.7469949Z + [[ inductor_huggingface_perf == *inductor_distributed* ]] 2023-03-08T02:51:16.7470432Z + [[ inductor_huggingface_perf == *dynamo* ]] 2023-03-08T02:51:16.7470726Z + [[ inductor_huggingface_perf == *dynamo* ]] 2023-03-08T02:51:16.7471026Z + [[ inductor_huggingface_perf == *huggingface* ]] 2023-03-08T02:51:16.7471281Z + install_torchvision 2023-03-08T02:51:16.7471502Z + local commit 2023-03-08T02:51:16.7471737Z ++ get_pinned_commit vision 2023-03-08T02:51:16.7471999Z ++ cat .github/ci_commit_pins/vision.txt 2023-03-08T02:51:16.7488209Z + commit=beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:16.7489180Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:16.7489923Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:17.1723127Z Collecting git+https://github.com/pytorch/vision.git@beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:17.1728801Z Cloning https://github.com/pytorch/vision.git (to revision beb4bb706b5e13009cb5d5586505c6d2896d184a) to /tmp/pip-req-build-fs7kzm39 2023-03-08T02:51:17.1756581Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-fs7kzm39 2023-03-08T02:51:19.5075368Z Running command git rev-parse -q --verify 'sha^beb4bb706b5e13009cb5d5586505c6d2896d184a' 2023-03-08T02:51:19.5100895Z Running command git fetch -q https://github.com/pytorch/vision.git beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:20.9160077Z Running command git checkout -q beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:21.4141658Z Resolved https://github.com/pytorch/vision.git to commit beb4bb706b5e13009cb5d5586505c6d2896d184a 2023-03-08T02:51:23.8164480Z Preparing metadata (setup.py) ... [?25l- done 2023-03-08T02:51:23.8246282Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.15.0a0+beb4bb7) (1.21.2) 2023-03-08T02:51:23.8250836Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.15.0a0+beb4bb7) (2.28.2) 2023-03-08T02:51:23.8255026Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.15.0a0+beb4bb7) (2.1.0a0+gitc88aa33) 2023-03-08T02:51:23.8262337Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.15.0a0+beb4bb7) (9.4.0) 2023-03-08T02:51:23.8464211Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.15.0a0+beb4bb7) (3.1.0) 2023-03-08T02:51:23.8470342Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.15.0a0+beb4bb7) (2022.12.7) 2023-03-08T02:51:23.8477005Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.15.0a0+beb4bb7) (1.26.14) 2023-03-08T02:51:23.8484242Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.15.0a0+beb4bb7) (3.4) 2023-03-08T02:51:23.8545608Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.15.0a0+beb4bb7) (3.1.2) 2023-03-08T02:51:23.8549772Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.15.0a0+beb4bb7) (1.11.1) 2023-03-08T02:51:23.8554035Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.15.0a0+beb4bb7) (4.5.0) 2023-03-08T02:51:23.8557872Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.15.0a0+beb4bb7) (3.9.0) 2023-03-08T02:51:23.8562032Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.15.0a0+beb4bb7) (2.6.3) 2023-03-08T02:51:23.8827598Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.15.0a0+beb4bb7) (2.1.2) 2023-03-08T02:51:23.9008476Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.15.0a0+beb4bb7) (1.3.0) 2023-03-08T02:51:23.9111516Z Building wheels for collected packages: torchvision 2023-03-08T02:52:52.2812228Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2023-03-08T02:52:52.2878597Z [?25h Created wheel for torchvision: filename=torchvision-0.15.0a0+beb4bb7-cp310-cp310-linux_x86_64.whl size=1889291 sha256=9e523d8017d5aa8644f56f9c64f65f6d48e0a658c91d921bc3c0ad594402540d 2023-03-08T02:52:52.2879941Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/69/53/d47c066a2e6bb87cb14170a74ba1c38f2c1a790507af88546e 2023-03-08T02:52:52.2939029Z Successfully built torchvision 2023-03-08T02:52:53.0404709Z Installing collected packages: torchvision 2023-03-08T02:52:53.5262623Z Successfully installed torchvision-0.15.0a0+beb4bb7 2023-03-08T02:52:53.6435378Z + install_huggingface 2023-03-08T02:52:53.6435643Z + local commit 2023-03-08T02:52:53.6439604Z ++ get_pinned_commit huggingface 2023-03-08T02:52:53.6439916Z ++ cat .github/ci_commit_pins/huggingface.txt 2023-03-08T02:52:53.6458230Z + commit=ebee0a27940adfbb30444d83387b9ea0f1173f40 2023-03-08T02:52:53.6458699Z + pip_install pandas 2023-03-08T02:52:53.6459239Z + pip install --progress-bar off pandas 2023-03-08T02:52:54.3153791Z Collecting pandas 2023-03-08T02:52:54.3793014Z Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB) 2023-03-08T02:52:54.6096174Z Requirement already satisfied: numpy>=1.21.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas) (1.21.2) 2023-03-08T02:52:54.6101557Z Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas) (2.8.2) 2023-03-08T02:52:54.6815139Z Collecting pytz>=2020.1 2023-03-08T02:52:54.6986689Z Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-03-08T02:52:54.7161287Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas) (1.16.0) 2023-03-08T02:52:55.5427475Z Installing collected packages: pytz, pandas 2023-03-08T02:52:59.5222745Z Successfully installed pandas-1.5.3 pytz-2022.7.1 2023-03-08T02:52:59.6117201Z + pip_install scipy 2023-03-08T02:52:59.6117841Z + pip install --progress-bar off scipy 2023-03-08T02:53:00.0291994Z Requirement already satisfied: scipy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (1.8.1) 2023-03-08T02:53:00.0310879Z Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scipy) (1.21.2) 2023-03-08T02:53:00.9371256Z + pip_install git+https://github.com/huggingface/transformers.git@ebee0a27940adfbb30444d83387b9ea0f1173f40#egg=transformers 2023-03-08T02:53:00.9372463Z + pip install --progress-bar off git+https://github.com/huggingface/transformers.git@ebee0a27940adfbb30444d83387b9ea0f1173f40#egg=transformers 2023-03-08T02:53:01.3586533Z Collecting transformers 2023-03-08T02:53:01.3589852Z Cloning https://github.com/huggingface/transformers.git (to revision ebee0a27940adfbb30444d83387b9ea0f1173f40) to /tmp/pip-install-nytkjz4x/transformers_158e64799d4a4d4daf04bf380b2462d0 2023-03-08T02:53:01.3611984Z Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-nytkjz4x/transformers_158e64799d4a4d4daf04bf380b2462d0 2023-03-08T02:53:07.9700322Z Running command git rev-parse -q --verify 'sha^ebee0a27940adfbb30444d83387b9ea0f1173f40' 2023-03-08T02:53:07.9726060Z Running command git fetch -q https://github.com/huggingface/transformers.git ebee0a27940adfbb30444d83387b9ea0f1173f40 2023-03-08T02:53:10.8932356Z Running command git checkout -q ebee0a27940adfbb30444d83387b9ea0f1173f40 2023-03-08T02:53:14.8447138Z Resolved https://github.com/huggingface/transformers.git to commit ebee0a27940adfbb30444d83387b9ea0f1173f40 2023-03-08T02:53:17.2529176Z Installing build dependencies ... [?25l- \ | / done 2023-03-08T02:53:17.5401464Z [?25h Getting requirements to build wheel ... [?25l- done 2023-03-08T02:53:17.9172312Z [?25h Preparing metadata (pyproject.toml) ... [?25l- done 2023-03-08T02:53:18.2626386Z [?25hRequirement already satisfied: tqdm>=4.27 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (4.65.0) 2023-03-08T02:53:18.3492911Z Collecting huggingface-hub<1.0,>=0.10.0 2023-03-08T02:53:18.4128664Z Downloading huggingface_hub-0.12.1-py3-none-any.whl (190 kB) 2023-03-08T02:53:18.4492315Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (3.9.0) 2023-03-08T02:53:19.0303014Z Collecting regex!=2019.12.17 2023-03-08T02:53:19.0491321Z Downloading regex-2022.10.31-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (770 kB) 2023-03-08T02:53:19.0847039Z Requirement already satisfied: numpy>=1.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (1.21.2) 2023-03-08T02:53:19.2692399Z Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 2023-03-08T02:53:19.2869115Z Downloading tokenizers-0.13.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB) 2023-03-08T02:53:19.3822667Z Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (6.0) 2023-03-08T02:53:19.3825004Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (2.28.2) 2023-03-08T02:53:19.3832524Z Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers) (23.0) 2023-03-08T02:53:19.4392777Z Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.5.0) 2023-03-08T02:53:19.4734737Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers) (3.1.0) 2023-03-08T02:53:19.4740842Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers) (3.4) 2023-03-08T02:53:19.4747408Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers) (2022.12.7) 2023-03-08T02:53:19.4756611Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers) (1.26.14) 2023-03-08T02:53:19.4953839Z Building wheels for collected packages: transformers 2023-03-08T02:53:21.2647044Z Building wheel for transformers (pyproject.toml) ... [?25l- \ | / - \ | / - \ | / done 2023-03-08T02:53:21.2815981Z [?25h Created wheel for transformers: filename=transformers-4.24.0.dev0-py3-none-any.whl size=5415588 sha256=a9748d82e14c5ad3504d6a09d194f51dcba681b0a97c7a6789d73fb45d445750 2023-03-08T02:53:21.2819775Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/c6/d8/16/ff2a102affc5219ad2aeb7dd1e1ce6742976ce69e190e29118 2023-03-08T02:53:21.2927975Z Successfully built transformers 2023-03-08T02:53:22.5056742Z Installing collected packages: tokenizers, regex, huggingface-hub, transformers 2023-03-08T02:53:25.8885474Z Successfully installed huggingface-hub-0.12.1 regex-2022.10.31 tokenizers-0.13.2 transformers-4.24.0.dev0 2023-03-08T02:53:26.1292136Z + test_dynamo_benchmark huggingface '' 2023-03-08T02:53:26.1292812Z + local suite=huggingface 2023-03-08T02:53:26.1293254Z + shift 2023-03-08T02:53:26.1293486Z + local shard_id= 2023-03-08T02:53:26.1296769Z + shift 2023-03-08T02:53:26.1297359Z + [[ inductor_huggingface_perf == *perf* ]] 2023-03-08T02:53:26.1298575Z + test_single_dynamo_benchmark amp huggingface '' --training --dtypes=amp 2023-03-08T02:53:26.1299090Z ++ pwd 2023-03-08T02:53:26.1300268Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2023-03-08T02:53:26.1301070Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2023-03-08T02:53:26.1323533Z + local name=amp 2023-03-08T02:53:26.1324338Z + shift 2023-03-08T02:53:26.1324777Z + local suite=huggingface 2023-03-08T02:53:26.1325234Z + shift 2023-03-08T02:53:26.1325602Z + local shard_id= 2023-03-08T02:53:26.1326153Z + shift 2023-03-08T02:53:26.1326769Z + partition_flags=() 2023-03-08T02:53:26.1327171Z + local partition_flags 2023-03-08T02:53:26.1327760Z + [[ -n 1 ]] 2023-03-08T02:53:26.1328183Z + [[ -n '' ]] 2023-03-08T02:53:26.1330086Z + [[ inductor_huggingface_perf == *perf* ]] 2023-03-08T02:53:26.1330424Z + MKL_THREADING_LAYER=GNU 2023-03-08T02:53:26.1331520Z + python benchmarks/dynamo/runner.py --suites=huggingface --base-sha=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e --output-dir=/var/lib/jenkins/workspace/test/test-reports --no-graphs --no-update-archive --no-gh-comment --training --dtypes=amp 2023-03-08T02:53:30.0195091Z 2023-03-08T02:53:30.0206635Z Downloading (…)lve/main/config.json: 0% 0.00/694 [00:00 2023-03-08T03:49:45.8772312Z self.optimizer_zero_grad(mod) 2023-03-08T03:49:45.8773593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T03:49:45.8774394Z pred = mod(**cloned_inputs) 2023-03-08T03:49:45.8775416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8775791Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8776359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T03:49:45.8776750Z outputs = self.longformer( 2023-03-08T03:49:45.8777250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8777600Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8778156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T03:49:45.8778553Z encoder_outputs = self.encoder( 2023-03-08T03:49:45.8779208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8779561Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8780119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T03:49:45.8780551Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T03:49:45.8781154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T03:49:45.8781558Z layer_outputs = layer_module( 2023-03-08T03:49:45.8782057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8782491Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8783037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T03:49:45.8783439Z self_attn_outputs = self.attention( 2023-03-08T03:49:45.8783948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8784309Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8784847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T03:49:45.8785232Z self_outputs = self.self( 2023-03-08T03:49:45.8785718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T03:49:45.8786064Z return forward_call(*args, **kwargs) 2023-03-08T03:49:45.8786616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T03:49:45.8787032Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T03:49:45.8787893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T03:49:45.8788328Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T03:49:45.8788971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T03:49:45.8789484Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T03:49:45.8790042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T03:49:45.8790416Z return callback(frame, cache_size, hooks) 2023-03-08T03:49:45.8790944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T03:49:45.8791454Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T03:49:45.8791963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T03:49:45.8792308Z return fn(*args, **kwargs) 2023-03-08T03:49:45.8792884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T03:49:45.8793227Z return _compile( 2023-03-08T03:49:45.8793699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T03:49:45.8794039Z r = func(*args, **kwargs) 2023-03-08T03:49:45.8794529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T03:49:45.8794900Z out_code = transform_code_object(code, transform) 2023-03-08T03:49:45.8795480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T03:49:45.8795985Z transformations(instructions, code_options) 2023-03-08T03:49:45.8796501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T03:49:45.8796835Z tracer.run() 2023-03-08T03:49:45.8797312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T03:49:45.8797650Z super().run() 2023-03-08T03:49:45.8798109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T03:49:45.8798451Z and self.step() 2023-03-08T03:49:45.8798928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T03:49:45.8799275Z getattr(self, inst.opname)(inst) 2023-03-08T03:49:45.8799787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T03:49:45.8800188Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T03:49:45.8800738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T03:49:45.8801150Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T03:49:45.8801736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T03:49:45.8802131Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T03:49:45.8802668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T03:49:45.8803005Z r = func(*args, **kwargs) 2023-03-08T03:49:45.8803509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T03:49:45.8803945Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T03:49:45.8804508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T03:49:45.8804916Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T03:49:45.8805489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T03:49:45.8805863Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T03:49:45.8806386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T03:49:45.8806750Z return compile_fx(*args, **kwargs) 2023-03-08T03:49:45.8807257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T03:49:45.8807591Z return aot_autograd( 2023-03-08T03:49:45.8808156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T03:49:45.8808563Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T03:49:45.8809120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T03:49:45.8809521Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T03:49:45.8810040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T03:49:45.8810377Z r = func(*args, **kwargs) 2023-03-08T03:49:45.8810903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T03:49:45.8811331Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T03:49:45.8811897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T03:49:45.8812310Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T03:49:45.8812976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T03:49:45.8813414Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T03:49:45.8813966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T03:49:45.8814286Z r = func(*args, **kwargs) 2023-03-08T03:49:45.8814782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T03:49:45.8815128Z return inner_compile( 2023-03-08T03:49:45.8815624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T03:49:45.8815990Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T03:49:45.8816503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T03:49:45.8816842Z return fn(*args, **kwargs) 2023-03-08T03:49:45.8817144Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T03:49:45.8817451Z return func(*args, **kwds) 2023-03-08T03:49:45.8817959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T03:49:45.8818309Z graph.run(*example_inputs) 2023-03-08T03:49:45.8818794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T03:49:45.8819127Z r = func(*args, **kwargs) 2023-03-08T03:49:45.8819591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T03:49:45.8819914Z return super().run(*args) 2023-03-08T03:49:45.8820394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T03:49:45.8820749Z self.env[node] = self.run_node(node) 2023-03-08T03:49:45.8821233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T03:49:45.8821579Z result = super().run_node(n) 2023-03-08T03:49:45.8822060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T03:49:45.8822484Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T03:49:45.8822995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T03:49:45.8823416Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T03:49:45.8823968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T03:49:45.8824318Z out = lowerings[target](*args, **kwargs) 2023-03-08T03:49:45.8824885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T03:49:45.8825231Z validate_ir(out) 2023-03-08T03:49:45.8825706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T03:49:45.8826044Z _check_tensorbox(node_or_nodes) 2023-03-08T03:49:45.8826541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T03:49:45.8826879Z assert isinstance( 2023-03-08T03:49:45.8827448Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T03:49:45.8828125Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T03:49:45.8828566Z target: aten._local_scalar_dense.default 2023-03-08T03:49:45.8828831Z args[0]: TensorBox(StorageBox( 2023-03-08T03:49:45.8829069Z Pointwise( 2023-03-08T03:49:45.8829315Z 'cpu', 2023-03-08T03:49:45.8829514Z torch.int64, 2023-03-08T03:49:45.8829829Z def inner_fn(index): 2023-03-08T03:49:45.8830094Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T03:49:45.8830384Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T03:49:45.8830653Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T03:49:45.8830901Z return tmp2 2023-03-08T03:49:45.8831107Z , 2023-03-08T03:49:45.8831318Z ranges=(), 2023-03-08T03:49:45.8831529Z origins={div} 2023-03-08T03:49:45.8831734Z ) 2023-03-08T03:49:45.8831904Z )) 2023-03-08T03:49:45.8832020Z 2023-03-08T03:49:45.8832025Z 2023-03-08T03:49:45.8832192Z You can suppress this exception and fall back to eager by setting: 2023-03-08T03:49:45.8832584Z torch._dynamo.config.suppress_errors = True 2023-03-08T03:49:45.8832770Z 2023-03-08T03:49:46.7869724Z ERROR 2023-03-08T03:49:57.4405823Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T03:50:00.4855077Z cuda train BartForCausalLM [2023-03-08 03:50:00,484] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:04.9886046Z [2023-03-08 03:50:04,987] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:08.6149721Z [2023-03-08 03:50:08,614] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:10.0107961Z [2023-03-08 03:50:10,010] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:11.4221598Z [2023-03-08 03:50:11,421] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:12.8399724Z [2023-03-08 03:50:12,839] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:14.2674715Z [2023-03-08 03:50:14,266] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:15.6896834Z [2023-03-08 03:50:15,688] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:17.1153264Z [2023-03-08 03:50:17,114] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:18.5552020Z [2023-03-08 03:50:18,554] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:19.9840531Z [2023-03-08 03:50:19,983] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:21.3991676Z [2023-03-08 03:50:21,398] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:22.8250666Z [2023-03-08 03:50:22,824] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:50:46.2716399Z 1.497x p=0.00 2023-03-08T03:51:05.7632387Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T03:51:39.4845849Z cuda train BartForConditionalGeneration [2023-03-08 03:51:39,482] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:52:33.8041328Z 1.481x p=0.00 2023-03-08T03:52:43.0814902Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T03:52:58.6598833Z cuda train BertForMaskedLM [2023-03-08 03:52:58,658] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:53:16.9007624Z [2023-03-08 03:53:16,899] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T03:53:26.8906759Z 1.599x p=0.00 2023-03-08T03:53:35.0950048Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T03:53:50.3790891Z cuda train BertForQuestionAnswering [2023-03-08 03:53:50,378] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:54:08.4151159Z [2023-03-08 03:54:08,414] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T03:54:17.0853785Z 1.765x p=0.00 2023-03-08T03:55:19.0790785Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T03:55:22.1801889Z cuda train BlenderbotForCausalLM [2023-03-08 03:55:22,179] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:24.0526023Z [2023-03-08 03:55:24,051] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:29.4546730Z [2023-03-08 03:55:29,453] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:30.8508291Z [2023-03-08 03:55:30,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:32.2431896Z [2023-03-08 03:55:32,242] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:33.6360065Z [2023-03-08 03:55:33,635] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:35.0342183Z [2023-03-08 03:55:35,033] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:36.4394411Z [2023-03-08 03:55:36,438] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:37.8382950Z [2023-03-08 03:55:37,837] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:39.2191872Z [2023-03-08 03:55:39,218] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:40.6116595Z [2023-03-08 03:55:40,611] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:42.0001974Z [2023-03-08 03:55:41,999] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:43.4122178Z [2023-03-08 03:55:43,411] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:44.8225811Z [2023-03-08 03:55:44,821] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:46.2373100Z [2023-03-08 03:55:46,236] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:47.6487012Z [2023-03-08 03:55:47,647] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:49.0632382Z [2023-03-08 03:55:49,062] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:50.4889287Z [2023-03-08 03:55:50,488] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:51.9058820Z [2023-03-08 03:55:51,905] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:53.3198855Z [2023-03-08 03:55:53,319] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:54.7278667Z [2023-03-08 03:55:54,727] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:56.1385197Z [2023-03-08 03:55:56,137] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:57.5636340Z [2023-03-08 03:55:57,562] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:55:58.9855311Z [2023-03-08 03:55:58,984] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:00.4255041Z [2023-03-08 03:56:00,424] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:22.6256783Z ERROR:common:Backend dynamo failed in warmup() 2023-03-08T03:56:22.6257319Z Traceback (most recent call last): 2023-03-08T03:56:22.6260946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1372, in warmup 2023-03-08T03:56:22.6261314Z fn(model, example_inputs) 2023-03-08T03:56:22.6262267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T03:56:22.6263017Z return fn(*args, **kwargs) 2023-03-08T03:56:22.6263383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T03:56:22.6264359Z cloned_inputs = clone_inputs(inputs) 2023-03-08T03:56:22.6265144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T03:56:22.6265533Z self.optimizer_zero_grad(mod) 2023-03-08T03:56:22.6265931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T03:56:22.6266889Z pred = mod(**cloned_inputs) 2023-03-08T03:56:22.6268266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 493, in 2023-03-08T03:56:22.6268707Z self.grad_scaler.scale(loss).backward() 2023-03-08T03:56:22.6269376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward 2023-03-08T03:56:22.6269725Z torch.autograd.backward( 2023-03-08T03:56:22.6270236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 204, in backward 2023-03-08T03:56:22.6270687Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-03-08T03:56:22.6271452Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 100.00 MiB (GPU 0; 39.45 GiB total capacity; 36.14 GiB already allocated; 63.25 MiB free; 38.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T03:56:23.7350825Z ERROR 2023-03-08T03:56:29.7281196Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T03:56:32.4385245Z cuda train BlenderbotSmallForCausalLM [2023-03-08 03:56:32,437] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:34.3730916Z [2023-03-08 03:56:34,372] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:38.6044265Z [2023-03-08 03:56:38,603] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:40.0233352Z [2023-03-08 03:56:40,022] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:41.4356879Z [2023-03-08 03:56:41,434] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:42.8493672Z [2023-03-08 03:56:42,848] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:44.2611813Z [2023-03-08 03:56:44,260] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:45.6856470Z [2023-03-08 03:56:45,684] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:56:47.1299923Z [2023-03-08 03:56:47,129] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:57:05.0931705Z 1.240x p=0.00 2023-03-08T03:57:13.9994311Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T03:57:37.0355693Z cuda train BlenderbotSmallForConditionalGeneration [2023-03-08 03:57:37,034] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:58:17.0182576Z 1.408x p=0.00 2023-03-08T03:58:25.8060408Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T03:58:41.4313203Z cuda train CamemBert [2023-03-08 03:58:41,429] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:58:59.1676470Z [2023-03-08 03:58:59,166] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T03:59:08.4245943Z 1.545x p=0.00 2023-03-08T03:59:17.4293488Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T03:59:20.8904438Z cuda train DebertaForMaskedLM [2023-03-08 03:59:20,889] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:59:26.9426501Z [2023-03-08 03:59:26,941] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T03:59:27.0085806Z [2023-03-08 03:59:27,008] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T03:59:28.3162600Z [2023-03-08 03:59:28,315] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:00:30.3868828Z 1.008x SAME 2023-03-08T04:00:38.6045904Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T04:00:42.2725251Z cuda train DebertaForQuestionAnswering [2023-03-08 04:00:42,271] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:00:48.0849809Z [2023-03-08 04:00:48,084] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T04:00:48.1561154Z [2023-03-08 04:00:48,155] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:00:49.4673536Z [2023-03-08 04:00:49,466] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:01:52.5450762Z 1.096x p=0.00 2023-03-08T04:02:20.1470293Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T04:02:26.3241490Z cuda train DebertaV2ForMaskedLM [2023-03-08 04:02:26,323] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:02:28.9354484Z [2023-03-08 04:02:28,934] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T04:02:28.9981156Z [2023-03-08 04:02:28,997] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:02:29.9782512Z [2023-03-08 04:02:29,977] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:03:24.3335450Z [2023-03-08 04:03:24,331] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T04:03:24.3336384Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T04:03:24.3336811Z reasons: ___check_obj_id(self, 140263934914912) 2023-03-08T04:03:24.3337423Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T04:04:35.0624286Z 0.927x p=0.00 2023-03-08T04:05:00.2370382Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T04:05:04.1189703Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 04:05:04,118] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:05:06.8213441Z [2023-03-08 04:05:06,820] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T04:05:06.8850091Z [2023-03-08 04:05:06,884] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:05:07.8873403Z [2023-03-08 04:05:07,886] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:06:02.7092384Z [2023-03-08 04:06:02,707] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T04:06:02.7093321Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T04:06:02.7093945Z reasons: ___check_obj_id(self, 140274580760960) 2023-03-08T04:06:02.7094764Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T04:07:14.9910538Z 0.882x p=0.00 2023-03-08T04:07:22.6137864Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T04:07:22.6138389Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T04:07:31.0546511Z cuda train DistilBertForMaskedLM [2023-03-08 04:07:31,053] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:07:58.6983114Z 1.217x p=0.00 2023-03-08T04:08:05.5742022Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T04:08:05.5742641Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T04:08:14.1646987Z cuda train DistilBertForQuestionAnswering [2023-03-08 04:08:14,163] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:08:43.6497696Z 1.432x p=0.00 2023-03-08T04:08:51.5978554Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T04:08:59.9182915Z cuda train DistillGPT2 [2023-03-08 04:08:59,917] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:09:20.5152887Z 1.661x p=0.00 2023-03-08T04:09:25.5750440Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T04:09:26.1591006Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T04:09:44.0074451Z cuda train ElectraForCausalLM [2023-03-08 04:09:44,006] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:10:20.0454068Z [2023-03-08 04:10:20,043] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T04:10:29.3469418Z 1.810x p=0.00 2023-03-08T04:10:35.3982224Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T04:10:50.9527582Z cuda train ElectraForQuestionAnswering [2023-03-08 04:10:50,951] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:11:13.9393431Z [2023-03-08 04:11:13,937] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T04:11:23.5526840Z 2.127x p=0.00 2023-03-08T04:11:32.5400095Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T04:11:46.4558007Z cuda train GPT2ForSequenceClassification [2023-03-08 04:11:46,454] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:12:11.7543397Z 2.280x p=0.00 2023-03-08T04:12:19.5998743Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T04:12:19.7583597Z Eager model failed to run 2023-03-08T04:12:19.7595611Z Traceback (most recent call last): 2023-03-08T04:12:19.7598898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T04:12:19.7599531Z self.model_iter_fn(model, example_inputs) 2023-03-08T04:12:19.7600172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T04:12:19.7600748Z pred = mod(**cloned_inputs) 2023-03-08T04:12:19.7602683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7605121Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7605884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T04:12:19.7606477Z outputs = self.fnet( 2023-03-08T04:12:19.7607024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7607394Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7607931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T04:12:19.7608582Z encoder_outputs = self.encoder( 2023-03-08T04:12:19.7609089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7609600Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7610162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T04:12:19.7610550Z layer_outputs = layer_module(hidden_states) 2023-03-08T04:12:19.7611119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7611489Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7612032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T04:12:19.7612776Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T04:12:19.7613541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7613935Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7614460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T04:12:19.7614849Z self_outputs = self.self(hidden_states) 2023-03-08T04:12:19.7615342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:12:19.7615703Z return forward_call(*args, **kwargs) 2023-03-08T04:12:19.7616229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T04:12:19.7616629Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T04:12:19.7617082Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T04:12:19.7617382Z 2023-03-08T04:12:19.7617552Z The above exception was the direct cause of the following exception: 2023-03-08T04:12:19.7617761Z 2023-03-08T04:12:19.7617879Z Traceback (most recent call last): 2023-03-08T04:12:19.7618206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T04:12:19.7618574Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T04:12:19.7618967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T04:12:19.7619327Z self.validate_model(model, example_inputs) 2023-03-08T04:12:19.7619681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T04:12:19.7620072Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T04:12:19.7620556Z NotImplementedError: Eager model failed to run 2023-03-08T04:12:19.7620755Z 2023-03-08T04:12:19.7620864Z WARNING:root:GoogleFnet failed to load 2023-03-08T04:12:27.4736792Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T04:12:43.5135104Z cuda train LayoutLMForMaskedLM [2023-03-08 04:12:43,512] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:13:12.3228286Z 1.582x p=0.00 2023-03-08T04:13:20.4126397Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T04:13:36.2619949Z cuda train LayoutLMForSequenceClassification [2023-03-08 04:13:36,260] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:14:09.7790155Z 1.772x p=0.00 2023-03-08T04:14:35.6215274Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T04:14:35.6215880Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T04:14:50.4209755Z cuda train M2M100ForConditionalGeneration [2023-03-08 04:14:50,419] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:07.9188940Z [2023-03-08 04:15:07,917] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:09.9518597Z [2023-03-08 04:15:09,951] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:16.8001561Z [2023-03-08 04:15:16,799] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:19.0627920Z [2023-03-08 04:15:19,061] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:21.3092875Z [2023-03-08 04:15:21,308] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:23.5803526Z [2023-03-08 04:15:23,579] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:25.8404090Z [2023-03-08 04:15:25,837] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:28.1129601Z [2023-03-08 04:15:28,112] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:30.3811166Z [2023-03-08 04:15:30,380] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:32.6591553Z [2023-03-08 04:15:32,658] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:34.9458672Z [2023-03-08 04:15:34,944] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:37.2010906Z [2023-03-08 04:15:37,200] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:15:39.4695768Z [2023-03-08 04:15:39,468] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:16:34.9940501Z 1.504x p=0.00 2023-03-08T04:16:47.5777804Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T04:16:50.6209422Z cuda train MBartForCausalLM [2023-03-08 04:16:50,620] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:16:55.0709793Z [2023-03-08 04:16:55,070] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:16:58.7014527Z [2023-03-08 04:16:58,700] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:00.0986615Z [2023-03-08 04:17:00,097] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:01.4760761Z [2023-03-08 04:17:01,475] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:02.8716114Z [2023-03-08 04:17:02,870] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:04.2724284Z [2023-03-08 04:17:04,271] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:05.6645323Z [2023-03-08 04:17:05,663] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:07.0429518Z [2023-03-08 04:17:07,042] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:08.4206202Z [2023-03-08 04:17:08,419] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:09.8124938Z [2023-03-08 04:17:09,811] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:11.2225179Z [2023-03-08 04:17:11,221] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:12.6248970Z [2023-03-08 04:17:12,624] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:17:36.5836338Z 1.517x p=0.00 2023-03-08T04:17:56.3160404Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T04:18:31.3557587Z cuda train MBartForConditionalGeneration [2023-03-08 04:18:31,354] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:19:31.6294866Z 1.485x p=0.00 2023-03-08T04:19:43.3417521Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T04:19:43.3418065Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T04:20:08.6831262Z cuda train MT5ForConditionalGeneration [2023-03-08 04:20:08,681] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:21:59.1204365Z 2.069x p=0.00 2023-03-08T04:22:05.1099435Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T04:22:13.2610258Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T04:22:42.8361312Z cuda train MegatronBertForCausalLM [2023-03-08 04:22:42,834] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:23:28.3384063Z 1.607x p=0.00 2023-03-08T04:23:42.1076537Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T04:24:11.0934952Z cuda train MegatronBertForQuestionAnswering [2023-03-08 04:24:11,091] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:24:59.5735645Z 1.628x p=0.00 2023-03-08T04:25:06.5589337Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T04:26:00.5012421Z cuda train MobileBertForMaskedLM [2023-03-08 04:26:00,499] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:27:40.6763713Z 2.439x p=0.00 2023-03-08T04:27:48.7062210Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T04:28:43.0668895Z cuda train MobileBertForQuestionAnswering [2023-03-08 04:28:43,065] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:17.2565576Z 2.158x p=0.00 2023-03-08T04:30:28.8672660Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T04:30:34.0096314Z cuda train OPTForCausalLM [2023-03-08 04:30:34,008] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:38.0228047Z [2023-03-08 04:30:38,021] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:39.3974560Z [2023-03-08 04:30:39,396] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:40.7538302Z [2023-03-08 04:30:40,753] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:42.1101671Z [2023-03-08 04:30:42,109] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:43.4693870Z [2023-03-08 04:30:43,468] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:44.8159939Z [2023-03-08 04:30:44,815] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:46.1714446Z [2023-03-08 04:30:46,170] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:47.5388138Z [2023-03-08 04:30:47,538] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:48.9121682Z [2023-03-08 04:30:48,911] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:50.2847329Z [2023-03-08 04:30:50,284] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:30:51.6815554Z [2023-03-08 04:30:51,680] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:16.4592183Z 2.457x p=0.00 2023-03-08T04:31:24.5284167Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T04:31:27.6987450Z cuda train PLBartForCausalLM [2023-03-08 04:31:27,697] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:32.1298280Z [2023-03-08 04:31:32,128] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:36.6812339Z [2023-03-08 04:31:36,680] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:38.1960338Z [2023-03-08 04:31:38,195] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:39.7430152Z [2023-03-08 04:31:39,742] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:41.2257345Z [2023-03-08 04:31:41,224] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:31:42.8064265Z [2023-03-08 04:31:42,805] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:00.0300489Z 1.608x p=0.00 2023-03-08T04:32:11.4828072Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T04:32:23.5988808Z cuda train PLBartForConditionalGeneration [2023-03-08 04:32:23,597] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:30.7121167Z [2023-03-08 04:32:30,711] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:34.9221545Z [2023-03-08 04:32:34,921] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:39.9620079Z [2023-03-08 04:32:39,961] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:42.2502379Z [2023-03-08 04:32:42,249] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:44.5547502Z [2023-03-08 04:32:44,554] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:46.8571797Z [2023-03-08 04:32:46,856] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:32:49.1700456Z [2023-03-08 04:32:49,169] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:15.6170229Z 1.594x p=0.00 2023-03-08T04:33:30.0125729Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T04:33:30.0126401Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T04:33:33.4403473Z cuda train PegasusForCausalLM [2023-03-08 04:33:33,439] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:35.0021063Z [2023-03-08 04:33:35,001] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:39.3932402Z [2023-03-08 04:33:39,392] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:40.7864600Z [2023-03-08 04:33:40,785] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:42.1721829Z [2023-03-08 04:33:42,171] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:43.5741259Z [2023-03-08 04:33:43,573] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:44.9638171Z [2023-03-08 04:33:44,963] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:46.3523570Z [2023-03-08 04:33:46,351] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:47.7627719Z [2023-03-08 04:33:47,761] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:49.1766423Z [2023-03-08 04:33:49,175] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:50.5850184Z [2023-03-08 04:33:50,584] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:52.0170900Z [2023-03-08 04:33:52,016] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:33:53.4516708Z [2023-03-08 04:33:53,450] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:34:16.7765827Z 1.184x p=0.00 2023-03-08T04:34:41.3232590Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T04:34:41.3233637Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T04:34:57.0410934Z cuda train PegasusForConditionalGeneration [2023-03-08 04:34:57,039] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:35:25.3657783Z [2023-03-08 04:35:25,364] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:36:09.6234261Z 1.330x p=0.00 2023-03-08T04:36:15.0784875Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T04:36:18.0877764Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T04:36:33.7862043Z cuda train RobertaForCausalLM [2023-03-08 04:36:33,785] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:36:53.0303893Z [2023-03-08 04:36:53,029] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T04:37:02.0082213Z 1.669x p=0.00 2023-03-08T04:37:10.2171107Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T04:37:25.5978133Z cuda train RobertaForQuestionAnswering [2023-03-08 04:37:25,596] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:37:43.1633826Z [2023-03-08 04:37:43,161] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T04:37:50.7052058Z 1.775x p=0.00 2023-03-08T04:37:56.7351842Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T04:37:56.7352349Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T04:38:00.1196081Z cuda train Speech2Text2ForCausalLM [2023-03-08 04:38:00,118] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:01.4645862Z [2023-03-08 04:38:01,463] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:05.7305255Z [2023-03-08 04:38:05,729] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:07.0951314Z [2023-03-08 04:38:07,094] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:08.4434794Z [2023-03-08 04:38:08,442] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:09.7953750Z [2023-03-08 04:38:09,794] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:11.1733475Z [2023-03-08 04:38:11,172] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:38:27.0824130Z 1.464x p=0.00 2023-03-08T04:38:33.7440960Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T04:38:51.7021972Z cuda train T5ForConditionalGeneration [2023-03-08 04:38:51,701] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:39:32.6594426Z 1.599x p=0.00 2023-03-08T04:39:39.7635675Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T04:39:57.5380916Z cuda train T5Small [2023-03-08 04:39:57,537] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:40:38.4387690Z 1.594x p=0.00 2023-03-08T04:40:50.3883111Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T04:40:53.6283365Z cuda train TrOCRForCausalLM [2023-03-08 04:40:53,627] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:40:58.1059985Z [2023-03-08 04:40:58,105] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:02.3616837Z [2023-03-08 04:41:02,360] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:03.7757352Z [2023-03-08 04:41:03,774] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:05.1853972Z [2023-03-08 04:41:05,184] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:06.5790766Z [2023-03-08 04:41:06,578] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:07.9898982Z [2023-03-08 04:41:07,989] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:09.4178243Z [2023-03-08 04:41:09,417] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:10.8363975Z [2023-03-08 04:41:10,835] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:12.2437148Z [2023-03-08 04:41:12,243] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:13.6365284Z [2023-03-08 04:41:13,635] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:15.0326929Z [2023-03-08 04:41:15,032] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:16.4283902Z [2023-03-08 04:41:16,427] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:41:40.5654821Z 1.241x p=0.00 2023-03-08T04:42:02.3769314Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T04:42:02.3770124Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T04:42:06.0200436Z cuda train XGLMForCausalLM [2023-03-08 04:42:06,019] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:07.4658765Z [2023-03-08 04:42:07,465] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:13.5925010Z [2023-03-08 04:42:13,591] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:15.0435630Z [2023-03-08 04:42:15,042] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:16.4946193Z [2023-03-08 04:42:16,493] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:17.9445360Z [2023-03-08 04:42:17,943] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:19.3974570Z [2023-03-08 04:42:19,396] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:20.8391031Z [2023-03-08 04:42:20,838] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:22.2793127Z [2023-03-08 04:42:22,278] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:23.7526526Z [2023-03-08 04:42:23,751] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:25.2208070Z [2023-03-08 04:42:25,220] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:26.6865812Z [2023-03-08 04:42:26,685] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:28.1435542Z [2023-03-08 04:42:28,142] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:29.6127725Z [2023-03-08 04:42:29,612] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:31.0792077Z [2023-03-08 04:42:31,078] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:32.5502368Z [2023-03-08 04:42:32,549] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:34.0229063Z [2023-03-08 04:42:34,022] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:35.5049327Z [2023-03-08 04:42:35,504] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:36.9787576Z [2023-03-08 04:42:36,978] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:38.4459666Z [2023-03-08 04:42:38,445] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:39.9158520Z [2023-03-08 04:42:39,915] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:41.3830767Z [2023-03-08 04:42:41,382] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:42.8508953Z [2023-03-08 04:42:42,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:44.3208232Z [2023-03-08 04:42:44,320] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:42:45.7926999Z [2023-03-08 04:42:45,792] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:43:22.7411320Z 1.510x p=0.00 2023-03-08T04:43:36.0524467Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T04:44:15.3385406Z cuda train XLNetLMHeadModel [2023-03-08 04:44:15,335] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:44:15.3586331Z [2023-03-08 04:44:15,358] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T04:44:56.7823553Z [2023-03-08 04:44:56,780] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T04:45:48.1018628Z 1.632x p=0.00 2023-03-08T04:45:57.5446384Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T04:46:23.8446733Z cuda train YituTechConvBert [2023-03-08 04:46:23,842] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:47:00.0848528Z [2023-03-08 04:47:00,083] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T04:47:18.1665013Z 1.543x p=0.00 2023-03-08T04:47:20.3506792Z speedup gmean=1.53x mean=1.565x 2023-03-08T04:47:20.3515483Z abs_latency gmean=nanx mean=77.741x 2023-03-08T04:47:20.3516936Z compilation_latency mean=60.639 seconds 2023-03-08T04:47:20.3517603Z compression_ratio mean=0.889x 2023-03-08T04:47:21.0633667Z + python benchmarks/dynamo/huggingface.py --performance --amp -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_performance.csv --training --inductor --disable-cudagraphs --no-skip --dashboard -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTJForCausalLM -x GPTNeoForSequenceClassification -x Reformer -x GPTNeoForCausalLM --cold-start-latency 2023-03-08T04:47:34.5229214Z WARNING:__main__:Running smaller batch size=4 for AlbertForMaskedLM, orig batch_size=8 2023-03-08T04:48:25.1120067Z cuda train AlbertForMaskedLM 1.649x p=0.00 2023-03-08T04:48:35.0853404Z WARNING:__main__:Running smaller batch size=4 for AlbertForQuestionAnswering, orig batch_size=8 2023-03-08T04:49:22.0473844Z cuda train AlbertForQuestionAnswering 1.659x p=0.00 2023-03-08T04:49:31.4543033Z WARNING:__main__:Running smaller batch size=4 for AllenaiLongformerBase, orig batch_size=8 2023-03-08T04:49:35.5861483Z cuda train AllenaiLongformerBase [2023-03-08 04:49:35,585] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:49:47.2340311Z ERROR:common:Backend dynamo failed in warmup() 2023-03-08T04:49:47.2340666Z Traceback (most recent call last): 2023-03-08T04:49:47.2341010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1372, in warmup 2023-03-08T04:49:47.2341339Z fn(model, example_inputs) 2023-03-08T04:49:47.2342067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T04:49:47.2342408Z return fn(*args, **kwargs) 2023-03-08T04:49:47.2346587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T04:49:47.2347491Z cloned_inputs = clone_inputs(inputs) 2023-03-08T04:49:47.2348300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T04:49:47.2349036Z self.optimizer_zero_grad(mod) 2023-03-08T04:49:47.2349553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T04:49:47.2349912Z pred = mod(**cloned_inputs) 2023-03-08T04:49:47.2350512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2350882Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2351436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T04:49:47.2351830Z outputs = self.longformer( 2023-03-08T04:49:47.2352338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2352700Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2353246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T04:49:47.2353638Z encoder_outputs = self.encoder( 2023-03-08T04:49:47.2354141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2354499Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2355043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T04:49:47.2355474Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T04:49:47.2356372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T04:49:47.2356778Z layer_outputs = layer_module( 2023-03-08T04:49:47.2357276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2357634Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2358248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T04:49:47.2358632Z self_attn_outputs = self.attention( 2023-03-08T04:49:47.2359138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2359503Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2360049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T04:49:47.2360432Z self_outputs = self.self( 2023-03-08T04:49:47.2360925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T04:49:47.2361411Z return forward_call(*args, **kwargs) 2023-03-08T04:49:47.2361958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T04:49:47.2362379Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T04:49:47.2362983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T04:49:47.2363410Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T04:49:47.2364042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T04:49:47.2364539Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T04:49:47.2365106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T04:49:47.2365470Z return callback(frame, cache_size, hooks) 2023-03-08T04:49:47.2366001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T04:49:47.2366394Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T04:49:47.2366903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T04:49:47.2367233Z return fn(*args, **kwargs) 2023-03-08T04:49:47.2367749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T04:49:47.2368142Z return _compile( 2023-03-08T04:49:47.2368611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T04:49:47.2368955Z r = func(*args, **kwargs) 2023-03-08T04:49:47.2369444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T04:49:47.2369828Z out_code = transform_code_object(code, transform) 2023-03-08T04:49:47.2370397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T04:49:47.2370813Z transformations(instructions, code_options) 2023-03-08T04:49:47.2371336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T04:49:47.2371662Z tracer.run() 2023-03-08T04:49:47.2372133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T04:49:47.2372464Z super().run() 2023-03-08T04:49:47.2372995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T04:49:47.2373338Z and self.step() 2023-03-08T04:49:47.2373818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T04:49:47.2374173Z getattr(self, inst.opname)(inst) 2023-03-08T04:49:47.2374666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T04:49:47.2375065Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T04:49:47.2375613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T04:49:47.2376030Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T04:49:47.2376621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T04:49:47.2377015Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T04:49:47.2377522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T04:49:47.2377918Z r = func(*args, **kwargs) 2023-03-08T04:49:47.2378463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T04:49:47.2378899Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T04:49:47.2379462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T04:49:47.2379876Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T04:49:47.2380422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T04:49:47.2380807Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T04:49:47.2381321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T04:49:47.2381689Z return compile_fx(*args, **kwargs) 2023-03-08T04:49:47.2382194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T04:49:47.2382539Z return aot_autograd( 2023-03-08T04:49:47.2383024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T04:49:47.2383426Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T04:49:47.2383998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T04:49:47.2384392Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T04:49:47.2384906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T04:49:47.2385249Z r = func(*args, **kwargs) 2023-03-08T04:49:47.2385800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T04:49:47.2386223Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T04:49:47.2386793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T04:49:47.2387401Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T04:49:47.2387956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T04:49:47.2388435Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T04:49:47.2388987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T04:49:47.2389323Z r = func(*args, **kwargs) 2023-03-08T04:49:47.2389900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T04:49:47.2390264Z return inner_compile( 2023-03-08T04:49:47.2390766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T04:49:47.2391137Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T04:49:47.2391644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T04:49:47.2391976Z return fn(*args, **kwargs) 2023-03-08T04:49:47.2392291Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T04:49:47.2392586Z return func(*args, **kwds) 2023-03-08T04:49:47.2393092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T04:49:47.2393448Z graph.run(*example_inputs) 2023-03-08T04:49:47.2393923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T04:49:47.2394332Z r = func(*args, **kwargs) 2023-03-08T04:49:47.2394847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T04:49:47.2395183Z return super().run(*args) 2023-03-08T04:49:47.2395646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T04:49:47.2395998Z self.env[node] = self.run_node(node) 2023-03-08T04:49:47.2396494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T04:49:47.2396829Z result = super().run_node(n) 2023-03-08T04:49:47.2397310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T04:49:47.2397691Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T04:49:47.2398260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T04:49:47.2398668Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T04:49:47.2399221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T04:49:47.2399590Z out = lowerings[target](*args, **kwargs) 2023-03-08T04:49:47.2400076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T04:49:47.2400412Z validate_ir(out) 2023-03-08T04:49:47.2400877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T04:49:47.2401227Z _check_tensorbox(node_or_nodes) 2023-03-08T04:49:47.2401710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T04:49:47.2402047Z assert isinstance( 2023-03-08T04:49:47.2402466Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T04:49:47.2403131Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T04:49:47.2403579Z target: aten._local_scalar_dense.default 2023-03-08T04:49:47.2403856Z args[0]: TensorBox(StorageBox( 2023-03-08T04:49:47.2404093Z Pointwise( 2023-03-08T04:49:47.2404327Z 'cpu', 2023-03-08T04:49:47.2404538Z torch.int64, 2023-03-08T04:49:47.2404761Z def inner_fn(index): 2023-03-08T04:49:47.2405014Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T04:49:47.2405308Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T04:49:47.2405585Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T04:49:47.2405815Z return tmp2 2023-03-08T04:49:47.2406021Z , 2023-03-08T04:49:47.2406215Z ranges=(), 2023-03-08T04:49:47.2406412Z origins={div} 2023-03-08T04:49:47.2406618Z ) 2023-03-08T04:49:47.2406800Z )) 2023-03-08T04:49:47.2406905Z 2023-03-08T04:49:47.2406994Z 2023-03-08T04:49:47.2407175Z You can suppress this exception and fall back to eager by setting: 2023-03-08T04:49:47.2407519Z torch._dynamo.config.suppress_errors = True 2023-03-08T04:49:47.2407704Z 2023-03-08T04:49:48.3808665Z ERROR 2023-03-08T04:49:59.0504572Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T04:50:02.0933334Z cuda train BartForCausalLM [2023-03-08 04:50:02,092] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:06.3693514Z [2023-03-08 04:50:06,368] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:10.0983064Z [2023-03-08 04:50:10,097] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:11.4076474Z [2023-03-08 04:50:11,406] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:12.6789004Z [2023-03-08 04:50:12,678] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:14.1255824Z [2023-03-08 04:50:14,124] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:15.4016323Z [2023-03-08 04:50:15,400] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:16.6649727Z [2023-03-08 04:50:16,664] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:18.0732819Z [2023-03-08 04:50:18,072] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:19.3335414Z [2023-03-08 04:50:19,332] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:20.5965951Z [2023-03-08 04:50:20,595] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:21.8766437Z [2023-03-08 04:50:21,875] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:23.3200358Z [2023-03-08 04:50:23,319] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:50:45.2240445Z 1.544x p=0.00 2023-03-08T04:51:04.9969509Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T04:51:40.3764857Z cuda train BartForConditionalGeneration [2023-03-08 04:51:40,374] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:52:36.8937997Z 1.482x p=0.00 2023-03-08T04:52:46.9301041Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T04:53:02.6334567Z cuda train BertForMaskedLM [2023-03-08 04:53:02,632] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:53:30.9336859Z 1.587x p=0.00 2023-03-08T04:53:38.9479771Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T04:53:54.3651684Z cuda train BertForQuestionAnswering [2023-03-08 04:53:54,364] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:54:21.2739066Z 1.749x p=0.00 2023-03-08T04:55:23.2229074Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T04:55:26.2655402Z cuda train BlenderbotForCausalLM [2023-03-08 04:55:26,264] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:27.9394429Z [2023-03-08 04:55:27,938] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:33.3749980Z [2023-03-08 04:55:33,374] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:34.6273377Z [2023-03-08 04:55:34,626] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:35.8839149Z [2023-03-08 04:55:35,883] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:37.2992044Z [2023-03-08 04:55:37,298] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:38.5550854Z [2023-03-08 04:55:38,554] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:39.8090175Z [2023-03-08 04:55:39,808] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:41.2194404Z [2023-03-08 04:55:41,218] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:42.4616786Z [2023-03-08 04:55:42,460] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:43.7038560Z [2023-03-08 04:55:43,703] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:44.9481831Z [2023-03-08 04:55:44,947] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:46.3659182Z [2023-03-08 04:55:46,365] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:47.6164691Z [2023-03-08 04:55:47,615] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:48.8614455Z [2023-03-08 04:55:48,860] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:50.2917836Z [2023-03-08 04:55:50,291] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:51.5413878Z [2023-03-08 04:55:51,540] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:52.7870610Z [2023-03-08 04:55:52,786] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:54.0240931Z [2023-03-08 04:55:54,023] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:55.4745600Z [2023-03-08 04:55:55,473] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:56.7216739Z [2023-03-08 04:55:56,721] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:57.9629165Z [2023-03-08 04:55:57,961] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:55:59.3825875Z [2023-03-08 04:55:59,381] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:00.6338393Z [2023-03-08 04:56:00,633] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:01.8885447Z [2023-03-08 04:56:01,887] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:37.8947626Z 1.054x p=0.00 2023-03-08T04:56:45.3290063Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T04:56:48.0539017Z cuda train BlenderbotSmallForCausalLM [2023-03-08 04:56:48,053] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:49.8078756Z [2023-03-08 04:56:49,807] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:54.0985437Z [2023-03-08 04:56:54,097] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:55.3750946Z [2023-03-08 04:56:55,374] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:56.6390668Z [2023-03-08 04:56:56,638] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:58.0414680Z [2023-03-08 04:56:58,040] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:56:59.3122246Z [2023-03-08 04:56:59,311] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:57:00.5810396Z [2023-03-08 04:57:00,580] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:57:01.8488364Z [2023-03-08 04:57:01,848] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:57:18.5921661Z 1.171x p=0.00 2023-03-08T04:57:27.5796394Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T04:57:50.8192573Z cuda train BlenderbotSmallForConditionalGeneration [2023-03-08 04:57:50,818] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:58:30.9779838Z 1.401x p=0.00 2023-03-08T04:58:40.1530234Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T04:58:56.1610759Z cuda train CamemBert [2023-03-08 04:58:56,160] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:59:23.6346636Z 1.534x p=0.00 2023-03-08T04:59:32.5588142Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T04:59:35.8854629Z cuda train DebertaForMaskedLM [2023-03-08 04:59:35,884] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:59:41.6639021Z [2023-03-08 04:59:41,663] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T04:59:42.6429382Z [2023-03-08 04:59:42,642] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:00:20.3918608Z 0.793x p=0.00 2023-03-08T05:00:28.7726806Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T05:00:32.1794198Z cuda train DebertaForQuestionAnswering [2023-03-08 05:00:32,178] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:00:37.5670875Z [2023-03-08 05:00:37,566] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:00:38.5198172Z [2023-03-08 05:00:38,519] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:01:16.6569646Z 0.987x p=0.05 2023-03-08T05:01:44.3134181Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T05:01:50.2917047Z cuda train DebertaV2ForMaskedLM [2023-03-08 05:01:50,290] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:01:52.5009564Z [2023-03-08 05:01:52,500] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:01:53.1236454Z [2023-03-08 05:01:53,122] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:02:25.6182645Z [2023-03-08 05:02:25,616] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T05:02:25.6184187Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T05:02:25.6184886Z reasons: ___check_obj_id(self, 140114627096928) 2023-03-08T05:02:25.6185577Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T05:02:58.1146899Z 0.627x p=0.00 2023-03-08T05:03:23.2756645Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T05:03:26.8325070Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 05:03:26,831] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:03:29.1148254Z [2023-03-08 05:03:29,114] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:03:29.7565947Z [2023-03-08 05:03:29,755] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:04:02.7980506Z [2023-03-08 05:04:02,796] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T05:04:02.7982348Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T05:04:02.7982784Z reasons: ___check_obj_id(self, 140600652687744) 2023-03-08T05:04:02.7983397Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T05:04:36.9839246Z 0.660x p=0.00 2023-03-08T05:04:44.5949637Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T05:04:44.5950509Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T05:04:53.1048886Z cuda train DistilBertForMaskedLM [2023-03-08 05:04:53,103] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:05:20.6518778Z 1.241x p=0.00 2023-03-08T05:05:27.4489359Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T05:05:27.4490119Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T05:05:36.1222290Z cuda train DistilBertForQuestionAnswering [2023-03-08 05:05:36,121] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:06:05.9059474Z 1.434x p=0.00 2023-03-08T05:06:13.8987492Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T05:06:22.2184063Z cuda train DistillGPT2 [2023-03-08 05:06:22,217] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:06:42.7693450Z 1.701x p=0.00 2023-03-08T05:06:47.8877344Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T05:06:48.4740749Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T05:07:06.2569441Z cuda train ElectraForCausalLM [2023-03-08 05:07:06,255] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:07:50.1936783Z 1.799x p=0.00 2023-03-08T05:07:56.2912331Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T05:08:12.0718476Z cuda train ElectraForQuestionAnswering [2023-03-08 05:08:12,070] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:08:44.9111482Z 2.094x p=0.00 2023-03-08T05:08:53.8793186Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T05:09:07.7951999Z cuda train GPT2ForSequenceClassification [2023-03-08 05:09:07,794] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:09:32.9783420Z 2.308x p=0.00 2023-03-08T05:09:41.1833875Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T05:09:41.3438092Z Eager model failed to run 2023-03-08T05:09:41.3451723Z Traceback (most recent call last): 2023-03-08T05:09:41.3456069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T05:09:41.3456697Z self.model_iter_fn(model, example_inputs) 2023-03-08T05:09:41.3457105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T05:09:41.3457468Z pred = mod(**cloned_inputs) 2023-03-08T05:09:41.3458847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3459239Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3459784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T05:09:41.3460152Z outputs = self.fnet( 2023-03-08T05:09:41.3460631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3461782Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3463048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T05:09:41.3463448Z encoder_outputs = self.encoder( 2023-03-08T05:09:41.3463973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3464332Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3464871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T05:09:41.3465272Z layer_outputs = layer_module(hidden_states) 2023-03-08T05:09:41.3465796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3466146Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3466680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T05:09:41.3467647Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T05:09:41.3468268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3468636Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3469166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T05:09:41.3469559Z self_outputs = self.self(hidden_states) 2023-03-08T05:09:41.3470054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:09:41.3470420Z return forward_call(*args, **kwargs) 2023-03-08T05:09:41.3470946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T05:09:41.3471348Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T05:09:41.3471799Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T05:09:41.3472093Z 2023-03-08T05:09:41.3472267Z The above exception was the direct cause of the following exception: 2023-03-08T05:09:41.3472475Z 2023-03-08T05:09:41.3472592Z Traceback (most recent call last): 2023-03-08T05:09:41.3472919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T05:09:41.3473291Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T05:09:41.3473685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T05:09:41.3474045Z self.validate_model(model, example_inputs) 2023-03-08T05:09:41.3474398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T05:09:41.3474790Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T05:09:41.3475131Z NotImplementedError: Eager model failed to run 2023-03-08T05:09:41.3475317Z 2023-03-08T05:09:41.3475425Z WARNING:root:GoogleFnet failed to load 2023-03-08T05:09:49.1500524Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T05:10:05.4269147Z cuda train LayoutLMForMaskedLM [2023-03-08 05:10:05,425] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:10:34.9628998Z 1.608x p=0.00 2023-03-08T05:10:43.3928995Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T05:10:59.2991111Z cuda train LayoutLMForSequenceClassification [2023-03-08 05:10:59,297] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:11:32.7188122Z 1.760x p=0.00 2023-03-08T05:11:58.4450003Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T05:11:58.4450627Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:12:13.1443502Z cuda train M2M100ForConditionalGeneration [2023-03-08 05:12:13,143] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:30.0634119Z [2023-03-08 05:12:30,062] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:32.1174151Z [2023-03-08 05:12:32,116] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:38.7620856Z [2023-03-08 05:12:38,761] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:40.8437344Z [2023-03-08 05:12:40,842] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:43.1991809Z [2023-03-08 05:12:43,198] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:45.2862307Z [2023-03-08 05:12:45,285] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:47.5749534Z [2023-03-08 05:12:47,573] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:49.6785046Z [2023-03-08 05:12:49,677] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:51.9632033Z [2023-03-08 05:12:51,962] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:54.0364191Z [2023-03-08 05:12:54,035] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:56.3297003Z [2023-03-08 05:12:56,329] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:12:58.4146242Z [2023-03-08 05:12:58,413] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:13:00.7351932Z [2023-03-08 05:13:00,734] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:13:53.6104872Z 1.224x p=0.00 2023-03-08T05:14:06.0986307Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T05:14:09.1323682Z cuda train MBartForCausalLM [2023-03-08 05:14:09,131] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:13.3822771Z [2023-03-08 05:14:13,381] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:17.0322261Z [2023-03-08 05:14:17,031] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:18.2821012Z [2023-03-08 05:14:18,281] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:19.5264555Z [2023-03-08 05:14:19,525] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:20.7680989Z [2023-03-08 05:14:20,767] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:22.1671999Z [2023-03-08 05:14:22,166] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:23.4157205Z [2023-03-08 05:14:23,414] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:24.6706687Z [2023-03-08 05:14:24,669] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:26.0745889Z [2023-03-08 05:14:26,073] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:27.3162814Z [2023-03-08 05:14:27,315] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:28.5535524Z [2023-03-08 05:14:28,552] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:29.9681446Z [2023-03-08 05:14:29,967] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:14:51.7164579Z 1.555x p=0.00 2023-03-08T05:15:11.3350436Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T05:15:45.4193390Z cuda train MBartForConditionalGeneration [2023-03-08 05:15:45,417] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:16:43.8853425Z 1.471x p=0.00 2023-03-08T05:16:55.4548459Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T05:16:55.4549003Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:17:20.9647461Z cuda train MT5ForConditionalGeneration [2023-03-08 05:17:20,962] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:19:11.3973651Z 1.763x p=0.00 2023-03-08T05:19:17.2646407Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T05:19:25.4105873Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T05:19:54.1950383Z cuda train MegatronBertForCausalLM [2023-03-08 05:19:54,193] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:20:40.7087080Z 1.430x p=0.00 2023-03-08T05:20:54.4962850Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T05:21:23.3288414Z cuda train MegatronBertForQuestionAnswering [2023-03-08 05:21:23,327] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:22:11.7491936Z 1.655x p=0.00 2023-03-08T05:22:18.9015500Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T05:23:13.5246371Z cuda train MobileBertForMaskedLM [2023-03-08 05:23:13,522] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:24:56.3200873Z 0.854x p=0.00 2023-03-08T05:25:04.6146720Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T05:25:58.4539101Z cuda train MobileBertForQuestionAnswering [2023-03-08 05:25:58,452] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:27:34.3555232Z 0.838x p=0.00 2023-03-08T05:27:46.4449746Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T05:27:51.4134356Z cuda train OPTForCausalLM [2023-03-08 05:27:51,412] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:27:55.3178999Z [2023-03-08 05:27:55,317] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:27:56.7654107Z [2023-03-08 05:27:56,764] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:27:57.9962192Z [2023-03-08 05:27:57,995] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:27:59.2372157Z [2023-03-08 05:27:59,236] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:00.6428536Z [2023-03-08 05:28:00,642] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:01.8866653Z [2023-03-08 05:28:01,885] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:03.1287045Z [2023-03-08 05:28:03,127] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:04.3672413Z [2023-03-08 05:28:04,366] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:05.7748843Z [2023-03-08 05:28:05,774] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:07.0131719Z [2023-03-08 05:28:07,012] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:08.2464922Z [2023-03-08 05:28:08,245] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:31.1216462Z 2.549x p=0.00 2023-03-08T05:28:39.3553615Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T05:28:42.5373377Z cuda train PLBartForCausalLM [2023-03-08 05:28:42,536] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:46.7963461Z [2023-03-08 05:28:46,795] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:51.3902385Z [2023-03-08 05:28:51,389] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:52.7327299Z [2023-03-08 05:28:52,731] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:54.0640519Z [2023-03-08 05:28:54,063] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:55.3783797Z [2023-03-08 05:28:55,377] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:28:56.8695650Z [2023-03-08 05:28:56,868] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:12.9803136Z 1.658x p=0.00 2023-03-08T05:29:24.3790736Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T05:29:36.7085804Z cuda train PLBartForConditionalGeneration [2023-03-08 05:29:36,707] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:43.6604283Z [2023-03-08 05:29:43,659] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:47.7172195Z [2023-03-08 05:29:47,716] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:52.6710751Z [2023-03-08 05:29:52,670] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:55.0484029Z [2023-03-08 05:29:55,047] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:57.2274803Z [2023-03-08 05:29:57,226] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:29:59.5866857Z [2023-03-08 05:29:59,585] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:01.7846074Z [2023-03-08 05:30:01,783] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:27.0026988Z 1.637x p=0.00 2023-03-08T05:30:41.7328329Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T05:30:41.7328854Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:30:45.0626021Z cuda train PegasusForCausalLM [2023-03-08 05:30:45,061] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:46.5542096Z [2023-03-08 05:30:46,553] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:50.8592345Z [2023-03-08 05:30:50,858] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:52.3447478Z [2023-03-08 05:30:52,343] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:53.6262243Z [2023-03-08 05:30:53,625] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:54.9044822Z [2023-03-08 05:30:54,903] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:56.3525672Z [2023-03-08 05:30:56,351] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:57.6504351Z [2023-03-08 05:30:57,649] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:30:58.9344213Z [2023-03-08 05:30:58,933] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:31:00.3993358Z [2023-03-08 05:31:00,398] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:31:01.6931240Z [2023-03-08 05:31:01,692] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:31:02.9937180Z [2023-03-08 05:31:02,992] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:31:04.2769878Z [2023-03-08 05:31:04,276] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:31:25.4300055Z 1.142x p=0.00 2023-03-08T05:31:49.8684045Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T05:31:49.8684646Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:32:05.2936813Z cuda train PegasusForConditionalGeneration [2023-03-08 05:32:05,292] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:32:32.7888631Z [2023-03-08 05:32:32,787] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:33:14.3297351Z 1.316x p=0.00 2023-03-08T05:33:20.1980455Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T05:33:23.2177412Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T05:33:39.2824537Z cuda train RobertaForCausalLM [2023-03-08 05:33:39,281] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:34:07.8855747Z 1.661x p=0.00 2023-03-08T05:34:16.1888516Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T05:34:31.7597882Z cuda train RobertaForQuestionAnswering [2023-03-08 05:34:31,758] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:34:57.4271724Z 1.758x p=0.00 2023-03-08T05:35:03.5763637Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T05:35:03.5764139Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:35:06.7839868Z cuda train Speech2Text2ForCausalLM [2023-03-08 05:35:06,783] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:08.0328211Z [2023-03-08 05:35:08,032] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:12.2547861Z [2023-03-08 05:35:12,253] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:13.6338961Z [2023-03-08 05:35:13,633] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:14.8642100Z [2023-03-08 05:35:14,863] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:16.0899704Z [2023-03-08 05:35:16,089] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:17.4961904Z [2023-03-08 05:35:17,495] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:35:32.3699921Z 1.532x p=0.00 2023-03-08T05:35:39.2637560Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T05:35:57.7571343Z cuda train T5ForConditionalGeneration [2023-03-08 05:35:57,755] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:36:38.5080083Z 1.718x p=0.00 2023-03-08T05:36:46.1510898Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T05:37:04.7582130Z cuda train T5Small [2023-03-08 05:37:04,756] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:37:45.7278815Z 1.719x p=0.00 2023-03-08T05:37:58.2762725Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T05:38:01.6318456Z cuda train TrOCRForCausalLM [2023-03-08 05:38:01,630] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:06.1050890Z [2023-03-08 05:38:06,104] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:10.4215404Z [2023-03-08 05:38:10,420] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:11.7295474Z [2023-03-08 05:38:11,728] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:13.0126696Z [2023-03-08 05:38:13,011] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:14.4829992Z [2023-03-08 05:38:14,482] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:15.7704554Z [2023-03-08 05:38:15,769] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:17.0686050Z [2023-03-08 05:38:17,067] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:18.5509986Z [2023-03-08 05:38:18,550] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:19.8561444Z [2023-03-08 05:38:19,855] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:21.1404635Z [2023-03-08 05:38:21,139] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:22.4259909Z [2023-03-08 05:38:22,425] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:23.8995791Z [2023-03-08 05:38:23,898] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:38:46.3480036Z 1.290x p=0.00 2023-03-08T05:39:08.3812300Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T05:39:08.3812777Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:39:11.8876004Z cuda train XGLMForCausalLM [2023-03-08 05:39:11,886] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:13.2583660Z [2023-03-08 05:39:13,257] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:19.5334642Z [2023-03-08 05:39:19,532] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:20.9097273Z [2023-03-08 05:39:20,908] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:22.2541055Z [2023-03-08 05:39:22,253] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:23.7706821Z [2023-03-08 05:39:23,769] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:25.1061450Z [2023-03-08 05:39:25,105] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:26.4462271Z [2023-03-08 05:39:26,445] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:27.7882927Z [2023-03-08 05:39:27,787] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:29.3238449Z [2023-03-08 05:39:29,323] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:30.6820557Z [2023-03-08 05:39:30,681] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:32.0358704Z [2023-03-08 05:39:32,035] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:33.5867056Z [2023-03-08 05:39:33,585] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:34.9355064Z [2023-03-08 05:39:34,934] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:36.2802804Z [2023-03-08 05:39:36,279] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:37.8278482Z [2023-03-08 05:39:37,827] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:39.1695364Z [2023-03-08 05:39:39,168] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:40.5103502Z [2023-03-08 05:39:40,509] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:42.0703174Z [2023-03-08 05:39:42,069] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:43.4264514Z [2023-03-08 05:39:43,425] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:44.7765787Z [2023-03-08 05:39:44,775] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:46.1079341Z [2023-03-08 05:39:46,107] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:47.6689149Z [2023-03-08 05:39:47,668] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:49.0199195Z [2023-03-08 05:39:49,019] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:39:50.3677728Z [2023-03-08 05:39:50,366] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:40:24.9082586Z 1.046x p=0.00 2023-03-08T05:40:38.5494323Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T05:41:17.4459979Z cuda train XLNetLMHeadModel [2023-03-08 05:41:17,444] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:41:17.4662632Z [2023-03-08 05:41:17,465] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T05:42:48.6179963Z 1.606x p=0.00 2023-03-08T05:42:57.8504594Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T05:43:24.8689426Z cuda train YituTechConvBert [2023-03-08 05:43:24,867] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T05:44:17.3570505Z 1.539x p=0.00 2023-03-08T05:44:19.2337050Z speedup gmean=1.45x mean=1.490x 2023-03-08T05:44:19.2345853Z abs_latency gmean=nanx mean=87.648x 2023-03-08T05:44:19.2346947Z compilation_latency mean=55.972 seconds 2023-03-08T05:44:19.2347626Z compression_ratio mean=1.038x 2023-03-08T05:44:19.7964328Z + python benchmarks/dynamo/huggingface.py --accuracy --amp -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/eager_huggingface_amp_training_cuda_accuracy.csv --training --backend=eager --no-skip --dashboard -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTJForCausalLM -x GPTNeoForSequenceClassification -x Reformer -x GPTNeoForCausalLM 2023-03-08T05:44:37.8262206Z cuda train AlbertForMaskedLM PASS 2023-03-08T05:44:52.2612991Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T05:45:16.3736213Z cuda train AllenaiLongformerBase PASS 2023-03-08T05:45:35.4761604Z cuda train BartForCausalLM PASS 2023-03-08T05:46:08.6903646Z cuda train BartForConditionalGeneration PASS 2023-03-08T05:46:22.8652660Z cuda train BertForMaskedLM PASS 2023-03-08T05:46:36.5992224Z cuda train BertForQuestionAnswering PASS 2023-03-08T05:47:38.6090895Z cuda train BlenderbotForCausalLM PASS 2023-03-08T05:47:50.5704841Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T05:48:08.6290900Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T05:48:22.7590310Z cuda train CamemBert PASS 2023-03-08T05:48:38.7068492Z cuda train DebertaForMaskedLM PASS 2023-03-08T05:48:54.0931085Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T05:49:21.9198492Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T05:50:02.6201331Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T05:50:09.7634167Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T05:50:12.8471799Z cuda train DistilBertForMaskedLM PASS 2023-03-08T05:50:19.2280415Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T05:50:22.3278431Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T05:50:32.9220683Z cuda train DistillGPT2 PASS 2023-03-08T05:50:37.6496717Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T05:50:47.0495118Z cuda train ElectraForCausalLM PASS 2023-03-08T05:50:58.5136543Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T05:51:15.1646355Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T05:51:22.5404028Z Eager model failed to run 2023-03-08T05:51:22.5412616Z Traceback (most recent call last): 2023-03-08T05:51:22.5413008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T05:51:22.5413372Z self.model_iter_fn(model, example_inputs) 2023-03-08T05:51:22.5416347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T05:51:22.5417055Z pred = mod(**cloned_inputs) 2023-03-08T05:51:22.5418479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5419112Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5420339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T05:51:22.5420956Z outputs = self.fnet( 2023-03-08T05:51:22.5421821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5422490Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5423506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T05:51:22.5424182Z encoder_outputs = self.encoder( 2023-03-08T05:51:22.5425093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5425684Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5426640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T05:51:22.5427524Z layer_outputs = layer_module(hidden_states) 2023-03-08T05:51:22.5428280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5428636Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5429167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T05:51:22.5429578Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T05:51:22.5430104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5430454Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5430977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T05:51:22.5431364Z self_outputs = self.self(hidden_states) 2023-03-08T05:51:22.5431859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T05:51:22.5432765Z return forward_call(*args, **kwargs) 2023-03-08T05:51:22.5433334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T05:51:22.5433751Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T05:51:22.5434192Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T05:51:22.5434483Z 2023-03-08T05:51:22.5434654Z The above exception was the direct cause of the following exception: 2023-03-08T05:51:22.5434864Z 2023-03-08T05:51:22.5434983Z Traceback (most recent call last): 2023-03-08T05:51:22.5435308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T05:51:22.5435681Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T05:51:22.5436078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T05:51:22.5436440Z self.validate_model(model, example_inputs) 2023-03-08T05:51:22.5436911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T05:51:22.5437302Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T05:51:22.5437644Z NotImplementedError: Eager model failed to run 2023-03-08T05:51:22.5437830Z 2023-03-08T05:51:22.5437941Z WARNING:root:GoogleFnet failed to load 2023-03-08T05:51:36.8197990Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T05:51:53.9015443Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T05:52:19.0978517Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:52:33.1414007Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T05:52:51.8876309Z cuda train MBartForCausalLM PASS 2023-03-08T05:53:24.7556830Z cuda train MBartForConditionalGeneration PASS 2023-03-08T05:53:35.1263030Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:53:45.0420239Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T05:53:50.0888883Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T05:54:10.2879491Z cuda train MegatronBertForCausalLM PASS 2023-03-08T05:54:34.9107593Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T05:54:59.1732009Z cuda train MobileBertForMaskedLM PASS 2023-03-08T05:55:23.2445818Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T05:55:39.3814582Z cuda train OPTForCausalLM PASS 2023-03-08T05:55:51.3927980Z cuda train PLBartForCausalLM PASS 2023-03-08T05:56:12.6601350Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T05:56:26.4674186Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:56:33.1225390Z cuda train PegasusForCausalLM PASS 2023-03-08T05:56:57.2085960Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T05:57:05.2298293Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T05:57:09.9991889Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T05:57:19.0848392Z cuda train RobertaForCausalLM PASS 2023-03-08T05:57:32.4926647Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T05:57:37.7919129Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:57:41.4863105Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T05:57:54.3932449Z cuda train T5ForConditionalGeneration PASS 2023-03-08T05:58:07.4786432Z cuda train T5Small PASS 2023-03-08T05:58:25.6490274Z cuda train TrOCRForCausalLM PASS 2023-03-08T05:58:46.9921529Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T05:58:58.0062407Z cuda train XGLMForCausalLM PASS 2023-03-08T05:59:23.4823193Z cuda train XLNetLMHeadModel PASS 2023-03-08T05:59:43.8382937Z cuda train YituTechConvBert PASS 2023-03-08T05:59:44.7086093Z accuracy pass_rate=95.56% 2023-03-08T05:59:44.7094288Z calls_captured gmean=442.18x mean=650.556x 2023-03-08T05:59:44.7102875Z unique_graphs gmean=3.51x mean=13.489x 2023-03-08T05:59:44.7110489Z graph_breaks gmean=9.98x mean=16.111x 2023-03-08T05:59:44.7117958Z unique_graph_breaks gmean=5.98x mean=6.733x 2023-03-08T05:59:45.2359258Z + python benchmarks/dynamo/huggingface.py --accuracy --amp -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/aot_eager_huggingface_amp_training_cuda_accuracy.csv --training --backend=aot_eager --no-skip --dashboard -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTJForCausalLM -x GPTNeoForSequenceClassification -x Reformer -x GPTNeoForCausalLM 2023-03-08T06:00:10.8797391Z cuda train AlbertForMaskedLM PASS 2023-03-08T06:00:32.9668613Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T06:01:24.7068764Z cuda train AllenaiLongformerBase PASS 2023-03-08T06:01:51.1683373Z cuda train BartForCausalLM PASS 2023-03-08T06:02:42.7454102Z cuda train BartForConditionalGeneration PASS 2023-03-08T06:03:04.0444205Z cuda train BertForMaskedLM PASS 2023-03-08T06:03:24.7129572Z cuda train BertForQuestionAnswering PASS 2023-03-08T06:04:26.3193759Z cuda train BlenderbotForCausalLM PASS 2023-03-08T06:04:42.9390033Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T06:05:13.0176726Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T06:05:34.1232498Z cuda train CamemBert PASS 2023-03-08T06:05:57.9977097Z cuda train DebertaForMaskedLM PASS 2023-03-08T06:06:21.2205265Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T06:06:48.9299443Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T06:07:44.4850545Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T06:07:51.6424795Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T06:07:58.3533159Z cuda train DistilBertForMaskedLM PASS 2023-03-08T06:08:04.6983263Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T06:08:11.3656471Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T06:08:25.1701008Z cuda train DistillGPT2 PASS 2023-03-08T06:08:29.9588335Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T06:08:46.7567762Z cuda train ElectraForCausalLM PASS 2023-03-08T06:09:05.2843576Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T06:09:28.0229330Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T06:09:35.4126192Z Eager model failed to run 2023-03-08T06:09:35.4136117Z Traceback (most recent call last): 2023-03-08T06:09:35.4136745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T06:09:35.4137234Z self.model_iter_fn(model, example_inputs) 2023-03-08T06:09:35.4137638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T06:09:35.4137995Z pred = mod(**cloned_inputs) 2023-03-08T06:09:35.4141977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4142521Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4143476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T06:09:35.4144511Z outputs = self.fnet( 2023-03-08T06:09:35.4145066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4145418Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4145946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T06:09:35.4146319Z encoder_outputs = self.encoder( 2023-03-08T06:09:35.4146809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4147387Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4148102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T06:09:35.4148503Z layer_outputs = layer_module(hidden_states) 2023-03-08T06:09:35.4149014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4149538Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4150074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T06:09:35.4150466Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T06:09:35.4150990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4151352Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4151910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T06:09:35.4152280Z self_outputs = self.self(hidden_states) 2023-03-08T06:09:35.4152792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:09:35.4153151Z return forward_call(*args, **kwargs) 2023-03-08T06:09:35.4153667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T06:09:35.4154078Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T06:09:35.4154522Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T06:09:35.4154814Z 2023-03-08T06:09:35.4154986Z The above exception was the direct cause of the following exception: 2023-03-08T06:09:35.4155196Z 2023-03-08T06:09:35.4155298Z Traceback (most recent call last): 2023-03-08T06:09:35.4155633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T06:09:35.4156003Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T06:09:35.4156394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T06:09:35.4156740Z self.validate_model(model, example_inputs) 2023-03-08T06:09:35.4157111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T06:09:35.4157497Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T06:09:35.4157864Z NotImplementedError: Eager model failed to run 2023-03-08T06:09:35.4158047Z 2023-03-08T06:09:35.4158166Z WARNING:root:GoogleFnet failed to load 2023-03-08T06:09:56.8019600Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T06:10:21.1100219Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T06:10:46.3511631Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:11:19.4462894Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T06:11:45.6164697Z cuda train MBartForCausalLM PASS 2023-03-08T06:12:36.8586558Z cuda train MBartForConditionalGeneration PASS 2023-03-08T06:12:47.1264101Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:13:09.1337588Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T06:13:14.2436659Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T06:13:48.3652452Z cuda train MegatronBertForCausalLM PASS 2023-03-08T06:14:26.8772492Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T06:15:21.9648291Z cuda train MobileBertForMaskedLM PASS 2023-03-08T06:16:17.2636227Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T06:16:40.9383874Z cuda train OPTForCausalLM PASS 2023-03-08T06:16:56.5745954Z cuda train PLBartForCausalLM PASS 2023-03-08T06:17:27.4954115Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T06:17:41.3024898Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T06:17:55.1906680Z cuda train PegasusForCausalLM PASS 2023-03-08T06:18:19.3396735Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:18:46.1557681Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T06:18:51.0820893Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T06:19:07.3193039Z cuda train RobertaForCausalLM PASS 2023-03-08T06:19:27.8142727Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T06:19:33.0800132Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T06:19:40.6201987Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T06:20:01.9058549Z cuda train T5ForConditionalGeneration PASS 2023-03-08T06:20:23.2848545Z cuda train T5Small PASS 2023-03-08T06:20:49.0279067Z cuda train TrOCRForCausalLM PASS 2023-03-08T06:21:10.3615091Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T06:21:36.1101526Z cuda train XGLMForCausalLM PASS 2023-03-08T06:22:22.5841180Z cuda train XLNetLMHeadModel PASS 2023-03-08T06:22:54.8539226Z cuda train YituTechConvBert PASS 2023-03-08T06:22:55.7901905Z accuracy pass_rate=95.56% 2023-03-08T06:22:55.7908854Z calls_captured gmean=442.18x mean=650.556x 2023-03-08T06:22:55.7916103Z unique_graphs gmean=3.51x mean=13.489x 2023-03-08T06:22:55.7922616Z graph_breaks gmean=9.98x mean=16.111x 2023-03-08T06:22:55.7929446Z unique_graph_breaks gmean=5.98x mean=6.733x 2023-03-08T06:22:56.2690724Z + python benchmarks/dynamo/huggingface.py --accuracy --amp -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_huggingface_amp_training_cuda_accuracy.csv --training --inductor --no-skip --dashboard -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTJForCausalLM -x GPTNeoForSequenceClassification -x Reformer -x GPTNeoForCausalLM --cold-start-latency 2023-03-08T06:23:44.1679978Z cuda train AlbertForMaskedLM PASS 2023-03-08T06:24:24.8562529Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T06:24:44.2708431Z cuda train AllenaiLongformerBase [2023-03-08 06:24:44,269] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T06:24:46.3918885Z [2023-03-08 06:24:46,391] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T06:24:51.2013060Z [2023-03-08 06:24:51,200] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T06:24:51.2385942Z ERROR:common:backend='inductor' raised: 2023-03-08T06:24:51.2386612Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T06:24:51.2387063Z target: aten._local_scalar_dense.default 2023-03-08T06:24:51.2392820Z args[0]: TensorBox(StorageBox( 2023-03-08T06:24:51.2393587Z Pointwise( 2023-03-08T06:24:51.2394196Z 'cpu', 2023-03-08T06:24:51.2394624Z torch.int64, 2023-03-08T06:24:51.2395052Z def inner_fn(index): 2023-03-08T06:24:51.2395534Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T06:24:51.2395973Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T06:24:51.2397123Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T06:24:51.2397460Z return tmp2 2023-03-08T06:24:51.2397796Z , 2023-03-08T06:24:51.2398120Z ranges=(), 2023-03-08T06:24:51.2398552Z origins={div} 2023-03-08T06:24:51.2427278Z ) 2023-03-08T06:24:51.2427749Z )) 2023-03-08T06:24:51.2427993Z 2023-03-08T06:24:51.2428006Z 2023-03-08T06:24:51.2428291Z You can suppress this exception and fall back to eager by setting: 2023-03-08T06:24:51.2428879Z torch._dynamo.config.suppress_errors = True 2023-03-08T06:24:51.2429181Z Traceback (most recent call last): 2023-03-08T06:24:51.2429544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T06:24:51.2430202Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T06:24:51.2430857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T06:24:51.2431202Z return fn(*args, **kwargs) 2023-03-08T06:24:51.2431532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T06:24:51.2431905Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T06:24:51.2432369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T06:24:51.2432729Z cloned_inputs = clone_inputs(inputs) 2023-03-08T06:24:51.2433124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T06:24:51.2433501Z self.optimizer_zero_grad(mod) 2023-03-08T06:24:51.2433895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T06:24:51.2434255Z pred = mod(**cloned_inputs) 2023-03-08T06:24:51.2434766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2435132Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2435685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T06:24:51.2436078Z outputs = self.longformer( 2023-03-08T06:24:51.2436588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2436953Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2437504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T06:24:51.2437903Z encoder_outputs = self.encoder( 2023-03-08T06:24:51.2438409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2438758Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2439314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T06:24:51.2439739Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T06:24:51.2440349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T06:24:51.2440735Z layer_outputs = layer_module( 2023-03-08T06:24:51.2441221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2441582Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2442235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T06:24:51.2442669Z self_attn_outputs = self.attention( 2023-03-08T06:24:51.2443188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2443547Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2444088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T06:24:51.2444470Z self_outputs = self.self( 2023-03-08T06:24:51.2444956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:24:51.2445313Z return forward_call(*args, **kwargs) 2023-03-08T06:24:51.2445854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T06:24:51.2446274Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T06:24:51.2446947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T06:24:51.2447369Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T06:24:51.2448005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T06:24:51.2448507Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T06:24:51.2449074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T06:24:51.2449428Z return callback(frame, cache_size, hooks) 2023-03-08T06:24:51.2449952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T06:24:51.2450339Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T06:24:51.2450846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T06:24:51.2451174Z return fn(*args, **kwargs) 2023-03-08T06:24:51.2451701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T06:24:51.2452052Z return _compile( 2023-03-08T06:24:51.2452544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:24:51.2452879Z r = func(*args, **kwargs) 2023-03-08T06:24:51.2453366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T06:24:51.2453731Z out_code = transform_code_object(code, transform) 2023-03-08T06:24:51.2454309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T06:24:51.2454731Z transformations(instructions, code_options) 2023-03-08T06:24:51.2455261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T06:24:51.2455581Z tracer.run() 2023-03-08T06:24:51.2456048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T06:24:51.2456379Z super().run() 2023-03-08T06:24:51.2456833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T06:24:51.2457163Z and self.step() 2023-03-08T06:24:51.2457637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T06:24:51.2457987Z getattr(self, inst.opname)(inst) 2023-03-08T06:24:51.2458540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T06:24:51.2458950Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T06:24:51.2459499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T06:24:51.2459910Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T06:24:51.2460499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T06:24:51.2460895Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T06:24:51.2461401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:24:51.2461724Z r = func(*args, **kwargs) 2023-03-08T06:24:51.2462227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T06:24:51.2462702Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T06:24:51.2463336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T06:24:51.2463752Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T06:24:51.2464293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T06:24:51.2464680Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T06:24:51.2465187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T06:24:51.2465548Z return compile_fx(*args, **kwargs) 2023-03-08T06:24:51.2466056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T06:24:51.2466389Z return aot_autograd( 2023-03-08T06:24:51.2466889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T06:24:51.2467746Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T06:24:51.2468347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T06:24:51.2468738Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T06:24:51.2469255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:24:51.2469595Z r = func(*args, **kwargs) 2023-03-08T06:24:51.2470191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T06:24:51.2470611Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T06:24:51.2471178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T06:24:51.2471595Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T06:24:51.2472157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T06:24:51.2472630Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T06:24:51.2473185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:24:51.2473521Z r = func(*args, **kwargs) 2023-03-08T06:24:51.2474002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T06:24:51.2474356Z return inner_compile( 2023-03-08T06:24:51.2474853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T06:24:51.2475234Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T06:24:51.2475851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T06:24:51.2476198Z return fn(*args, **kwargs) 2023-03-08T06:24:51.2476514Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T06:24:51.2476808Z return func(*args, **kwds) 2023-03-08T06:24:51.2477323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T06:24:51.2477683Z graph.run(*example_inputs) 2023-03-08T06:24:51.2478169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:24:51.2478492Z r = func(*args, **kwargs) 2023-03-08T06:24:51.2478965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T06:24:51.2479299Z return super().run(*args) 2023-03-08T06:24:51.2479757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T06:24:51.2480109Z self.env[node] = self.run_node(node) 2023-03-08T06:24:51.2480676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T06:24:51.2481006Z result = super().run_node(n) 2023-03-08T06:24:51.2481488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T06:24:51.2481864Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T06:24:51.2482421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T06:24:51.2482828Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T06:24:51.2483380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T06:24:51.2483746Z out = lowerings[target](*args, **kwargs) 2023-03-08T06:24:51.2484239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T06:24:51.2484575Z validate_ir(out) 2023-03-08T06:24:51.2485047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T06:24:51.2485397Z _check_tensorbox(node_or_nodes) 2023-03-08T06:24:51.2485877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T06:24:51.2486214Z assert isinstance( 2023-03-08T06:24:51.2486630Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T06:24:51.2487293Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T06:24:51.2487733Z target: aten._local_scalar_dense.default 2023-03-08T06:24:51.2488008Z args[0]: TensorBox(StorageBox( 2023-03-08T06:24:51.2488235Z Pointwise( 2023-03-08T06:24:51.2488483Z 'cpu', 2023-03-08T06:24:51.2488693Z torch.int64, 2023-03-08T06:24:51.2488907Z def inner_fn(index): 2023-03-08T06:24:51.2489175Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T06:24:51.2489467Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T06:24:51.2489748Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T06:24:51.2489976Z return tmp2 2023-03-08T06:24:51.2490187Z , 2023-03-08T06:24:51.2490384Z ranges=(), 2023-03-08T06:24:51.2490579Z origins={div} 2023-03-08T06:24:51.2490779Z ) 2023-03-08T06:24:51.2490959Z )) 2023-03-08T06:24:51.2491062Z 2023-03-08T06:24:51.2491068Z 2023-03-08T06:24:51.2491236Z You can suppress this exception and fall back to eager by setting: 2023-03-08T06:24:51.2491580Z torch._dynamo.config.suppress_errors = True 2023-03-08T06:24:51.2491762Z 2023-03-08T06:24:51.2491934Z TorchDynamo optimized model failed to run because of following error 2023-03-08T06:24:51.2498477Z FAIL 2023-03-08T06:25:40.0498454Z cuda train BartForCausalLM PASS 2023-03-08T06:27:06.5549055Z cuda train BartForConditionalGeneration PASS 2023-03-08T06:27:43.4939850Z cuda train BertForMaskedLM PASS 2023-03-08T06:28:18.0005829Z cuda train BertForQuestionAnswering PASS 2023-03-08T06:29:20.4449908Z cuda train BlenderbotForCausalLM PASS 2023-03-08T06:30:05.9051944Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T06:31:19.3647346Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T06:31:56.0139707Z cuda train CamemBert PASS 2023-03-08T06:32:12.8872832Z cuda train DebertaForMaskedLM [2023-03-08 06:32:12,886] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T06:32:53.8967118Z PASS 2023-03-08T06:33:09.9902097Z cuda train DebertaForQuestionAnswering [2023-03-08 06:33:09,989] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T06:33:49.1433357Z PASS 2023-03-08T06:34:17.2233452Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T06:34:50.6977945Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 06:34:50,696] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T06:35:45.5471941Z ERROR:common:CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 39.45 GiB total capacity; 36.17 GiB already allocated; 15.25 MiB free; 38.80 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T06:35:45.5473093Z Traceback (most recent call last): 2023-03-08T06:35:45.5473808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T06:35:45.5474305Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T06:35:45.5475287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T06:35:45.5475827Z return fn(*args, **kwargs) 2023-03-08T06:35:45.5476318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T06:35:45.5476683Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T06:35:45.5477097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T06:35:45.5477480Z cloned_inputs = clone_inputs(inputs) 2023-03-08T06:35:45.5477939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T06:35:45.5478299Z self.optimizer_zero_grad(mod) 2023-03-08T06:35:45.5478686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T06:35:45.5479054Z pred = mod(**cloned_inputs) 2023-03-08T06:35:45.5479424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 493, in 2023-03-08T06:35:45.5480348Z self.grad_scaler.scale(loss).backward() 2023-03-08T06:35:45.5482052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward 2023-03-08T06:35:45.5482402Z torch.autograd.backward( 2023-03-08T06:35:45.5482922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 204, in backward 2023-03-08T06:35:45.5483370Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-03-08T06:35:45.5483938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply 2023-03-08T06:35:45.5484283Z return user_fn(self, *args) 2023-03-08T06:35:45.5484768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2361, in backward 2023-03-08T06:35:45.5485129Z out = call_compiled_backward() 2023-03-08T06:35:45.5485983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2332, in call_compiled_backward 2023-03-08T06:35:45.5486421Z CompiledFunction.compiled_bw = aot_config.bw_compiler( 2023-03-08T06:35:45.5487005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 38, in _wrapped_bw_compiler 2023-03-08T06:35:45.5487453Z return eval_frame.disable(eval_frame.disable(bw_compiler)(*args, **kwargs)) 2023-03-08T06:35:45.5488061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T06:35:45.5488386Z return fn(*args, **kwargs) 2023-03-08T06:35:45.5488865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:35:45.5489200Z r = func(*args, **kwargs) 2023-03-08T06:35:45.5489682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 473, in bw_compiler 2023-03-08T06:35:45.5490157Z return inner_compile( 2023-03-08T06:35:45.5490659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T06:35:45.5491045Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T06:35:45.5491533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T06:35:45.5491869Z return fn(*args, **kwargs) 2023-03-08T06:35:45.5492183Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T06:35:45.5492472Z return func(*args, **kwds) 2023-03-08T06:35:45.5492973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 193, in compile_fx_inner 2023-03-08T06:35:45.5493337Z compiled_fn = cudagraphify( 2023-03-08T06:35:45.5493825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T06:35:45.5494147Z r = func(*args, **kwargs) 2023-03-08T06:35:45.5494645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 261, in cudagraphify 2023-03-08T06:35:45.5495055Z return cudagraphify_impl(model, inputs, static_input_idxs) 2023-03-08T06:35:45.5495597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 324, in cudagraphify_impl 2023-03-08T06:35:45.5495959Z model(list(static_inputs)) 2023-03-08T06:35:45.5496363Z File "/tmp/tmpmbw8b4_3/xy/cxy7of46lzkw3hroxjf7sitdhbew3aswm7sic3agjqr5cr2lcp4h.py", line 408, in call 2023-03-08T06:35:45.5496931Z buf19 = empty_strided((6144, 1536), (1536, 1), device='cuda', dtype=torch.float32) 2023-03-08T06:35:45.5497648Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 39.45 GiB total capacity; 36.17 GiB already allocated; 15.25 MiB free; 38.80 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T06:35:45.5498402Z TorchDynamo optimized model failed to run because of following error 2023-03-08T06:35:45.5580783Z FAIL 2023-03-08T06:35:52.8406944Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T06:36:29.1210385Z cuda train DistilBertForMaskedLM PASS 2023-03-08T06:36:35.9271084Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T06:37:02.8075856Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T06:37:25.4205704Z cuda train DistillGPT2 PASS 2023-03-08T06:37:30.5496008Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T06:38:14.5182050Z cuda train ElectraForCausalLM PASS 2023-03-08T06:38:51.9096993Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T06:39:29.0779826Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T06:39:36.9384058Z Eager model failed to run 2023-03-08T06:39:36.9394758Z Traceback (most recent call last): 2023-03-08T06:39:36.9395403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T06:39:36.9395778Z self.model_iter_fn(model, example_inputs) 2023-03-08T06:39:36.9396165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T06:39:36.9398469Z pred = mod(**cloned_inputs) 2023-03-08T06:39:36.9399270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9399661Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9400211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T06:39:36.9400881Z outputs = self.fnet( 2023-03-08T06:39:36.9401468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9402262Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9402889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T06:39:36.9403260Z encoder_outputs = self.encoder( 2023-03-08T06:39:36.9403876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9404366Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9404891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T06:39:36.9405293Z layer_outputs = layer_module(hidden_states) 2023-03-08T06:39:36.9405997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9406607Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9407142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T06:39:36.9407552Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T06:39:36.9408081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9408444Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9408961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T06:39:36.9409564Z self_outputs = self.self(hidden_states) 2023-03-08T06:39:36.9410680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T06:39:36.9411083Z return forward_call(*args, **kwargs) 2023-03-08T06:39:36.9411632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T06:39:36.9412046Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T06:39:36.9412566Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T06:39:36.9412846Z 2023-03-08T06:39:36.9413020Z The above exception was the direct cause of the following exception: 2023-03-08T06:39:36.9413250Z 2023-03-08T06:39:36.9413368Z Traceback (most recent call last): 2023-03-08T06:39:36.9413713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T06:39:36.9414087Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T06:39:36.9414465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T06:39:36.9414827Z self.validate_model(model, example_inputs) 2023-03-08T06:39:36.9415426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T06:39:36.9415815Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T06:39:36.9416162Z NotImplementedError: Eager model failed to run 2023-03-08T06:39:36.9416345Z 2023-03-08T06:39:36.9416470Z WARNING:root:GoogleFnet failed to load 2023-03-08T06:40:10.5418581Z cuda train LayoutLMForMaskedLM [2023-03-08 06:40:10,540] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T06:40:13.5111994Z PASS 2023-03-08T06:40:50.7914232Z cuda train LayoutLMForSequenceClassification [2023-03-08 06:40:50,790] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T06:40:52.9482224Z PASS 2023-03-08T06:41:18.5347825Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:42:46.8145254Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T06:43:36.2118050Z cuda train MBartForCausalLM PASS 2023-03-08T06:45:06.2166764Z cuda train MBartForConditionalGeneration PASS 2023-03-08T06:45:17.4864386Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:46:22.0755253Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T06:46:27.5466075Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T06:47:29.1348712Z cuda train MegatronBertForCausalLM PASS 2023-03-08T06:48:35.4452744Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T06:50:37.5666205Z cuda train MobileBertForMaskedLM PASS 2023-03-08T06:52:31.0027933Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T06:53:16.0505640Z cuda train OPTForCausalLM PASS 2023-03-08T06:53:48.5948969Z cuda train PLBartForCausalLM PASS 2023-03-08T06:54:47.7499364Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T06:55:02.0424905Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T06:55:46.4171395Z cuda train PegasusForCausalLM PASS 2023-03-08T06:56:10.7964168Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T06:57:33.7134683Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T06:57:39.0647039Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T06:58:10.3230213Z cuda train RobertaForCausalLM PASS 2023-03-08T06:58:45.0830679Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T06:58:50.8007755Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T06:59:23.3408536Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T07:00:03.8169432Z cuda train T5ForConditionalGeneration PASS 2023-03-08T07:00:46.5908938Z cuda train T5Small PASS 2023-03-08T07:01:35.6650098Z cuda train TrOCRForCausalLM PASS 2023-03-08T07:01:57.3095719Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T07:03:07.2559829Z cuda train XGLMForCausalLM PASS 2023-03-08T07:03:55.3230478Z cuda train XLNetLMHeadModel [2023-03-08 07:03:55,321] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T07:04:12.0230860Z [2023-03-08 07:04:12,022] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T07:04:32.6814236Z PASS 2023-03-08T07:05:45.4812110Z cuda train YituTechConvBert PASS 2023-03-08T07:05:47.0003654Z accuracy pass_rate=91.11% 2023-03-08T07:05:47.0011394Z calls_captured gmean=414.53x mean=614.489x 2023-03-08T07:05:47.0023251Z unique_graphs gmean=3.30x mean=10.133x 2023-03-08T07:05:47.0033066Z graph_breaks gmean=9.51x mean=13.422x 2023-03-08T07:05:47.0042278Z unique_graph_breaks gmean=5.85x mean=6.556x 2023-03-08T07:05:47.4989767Z + python benchmarks/dynamo/huggingface.py --accuracy --amp -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_accuracy.csv --training --inductor --disable-cudagraphs --no-skip --dashboard -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTJForCausalLM -x GPTNeoForSequenceClassification -x Reformer -x GPTNeoForCausalLM --cold-start-latency 2023-03-08T07:06:34.3997672Z cuda train AlbertForMaskedLM PASS 2023-03-08T07:07:14.5744052Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T07:07:40.1202317Z cuda train AllenaiLongformerBase ERROR:common:backend='inductor' raised: 2023-03-08T07:07:40.1203190Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T07:07:40.1203670Z target: aten._local_scalar_dense.default 2023-03-08T07:07:40.1204230Z args[0]: TensorBox(StorageBox( 2023-03-08T07:07:40.1204563Z Pointwise( 2023-03-08T07:07:40.1204964Z 'cpu', 2023-03-08T07:07:40.1205181Z torch.int64, 2023-03-08T07:07:40.1205409Z def inner_fn(index): 2023-03-08T07:07:40.1205661Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T07:07:40.1206924Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T07:07:40.1207457Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T07:07:40.1207969Z return tmp2 2023-03-08T07:07:40.1208285Z , 2023-03-08T07:07:40.1208559Z ranges=(), 2023-03-08T07:07:40.1208878Z origins={div} 2023-03-08T07:07:40.1209172Z ) 2023-03-08T07:07:40.1209430Z )) 2023-03-08T07:07:40.1209631Z 2023-03-08T07:07:40.1209642Z 2023-03-08T07:07:40.1209957Z You can suppress this exception and fall back to eager by setting: 2023-03-08T07:07:40.1210609Z torch._dynamo.config.suppress_errors = True 2023-03-08T07:07:40.1211153Z Traceback (most recent call last): 2023-03-08T07:07:40.1211794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T07:07:40.1212338Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T07:07:40.1213027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T07:07:40.1213359Z return fn(*args, **kwargs) 2023-03-08T07:07:40.1213704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T07:07:40.1214086Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T07:07:40.1214475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T07:07:40.1214845Z cloned_inputs = clone_inputs(inputs) 2023-03-08T07:07:40.1215243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T07:07:40.1215619Z self.optimizer_zero_grad(mod) 2023-03-08T07:07:40.1215990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T07:07:40.1216350Z pred = mod(**cloned_inputs) 2023-03-08T07:07:40.1216857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1217210Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1217846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T07:07:40.1218238Z outputs = self.longformer( 2023-03-08T07:07:40.1218741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1219090Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1219884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T07:07:40.1220293Z encoder_outputs = self.encoder( 2023-03-08T07:07:40.1220787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1221150Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1221705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T07:07:40.1222140Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T07:07:40.1222746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T07:07:40.1223146Z layer_outputs = layer_module( 2023-03-08T07:07:40.1223639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1223999Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1224630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T07:07:40.1225023Z self_attn_outputs = self.attention( 2023-03-08T07:07:40.1225530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1225874Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1226424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T07:07:40.1226802Z self_outputs = self.self( 2023-03-08T07:07:40.1227725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:07:40.1228098Z return forward_call(*args, **kwargs) 2023-03-08T07:07:40.1228658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T07:07:40.1229087Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T07:07:40.1229694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T07:07:40.1230119Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T07:07:40.1230755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T07:07:40.1231259Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T07:07:40.1231827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T07:07:40.1232187Z return callback(frame, cache_size, hooks) 2023-03-08T07:07:40.1232710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T07:07:40.1233101Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T07:07:40.1233609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T07:07:40.1233934Z return fn(*args, **kwargs) 2023-03-08T07:07:40.1234448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T07:07:40.1234798Z return _compile( 2023-03-08T07:07:40.1235257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T07:07:40.1235594Z r = func(*args, **kwargs) 2023-03-08T07:07:40.1236080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T07:07:40.1236573Z out_code = transform_code_object(code, transform) 2023-03-08T07:07:40.1237167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T07:07:40.1237623Z transformations(instructions, code_options) 2023-03-08T07:07:40.1238152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T07:07:40.1238477Z tracer.run() 2023-03-08T07:07:40.1238954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T07:07:40.1239286Z super().run() 2023-03-08T07:07:40.1239746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T07:07:40.1240080Z and self.step() 2023-03-08T07:07:40.1240553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T07:07:40.1240916Z getattr(self, inst.opname)(inst) 2023-03-08T07:07:40.1241492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T07:07:40.1241895Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T07:07:40.1242440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T07:07:40.1242855Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T07:07:40.1243449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T07:07:40.1243842Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T07:07:40.1244343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T07:07:40.1244667Z r = func(*args, **kwargs) 2023-03-08T07:07:40.1245174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T07:07:40.1245613Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T07:07:40.1246174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T07:07:40.1246584Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T07:07:40.1247130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T07:07:40.1247515Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T07:07:40.1248065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T07:07:40.1248431Z return compile_fx(*args, **kwargs) 2023-03-08T07:07:40.1248939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T07:07:40.1249288Z return aot_autograd( 2023-03-08T07:07:40.1249775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T07:07:40.1250174Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T07:07:40.1250737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T07:07:40.1251127Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T07:07:40.1251638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T07:07:40.1251973Z r = func(*args, **kwargs) 2023-03-08T07:07:40.1252502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T07:07:40.1252932Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T07:07:40.1253558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T07:07:40.1253978Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T07:07:40.1254553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T07:07:40.1254995Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T07:07:40.1255542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T07:07:40.1255876Z r = func(*args, **kwargs) 2023-03-08T07:07:40.1256351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T07:07:40.1256698Z return inner_compile( 2023-03-08T07:07:40.1257192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T07:07:40.1257563Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T07:07:40.1258186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T07:07:40.1258527Z return fn(*args, **kwargs) 2023-03-08T07:07:40.1258845Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T07:07:40.1259136Z return func(*args, **kwds) 2023-03-08T07:07:40.1259644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T07:07:40.1260005Z graph.run(*example_inputs) 2023-03-08T07:07:40.1260479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T07:07:40.1260818Z r = func(*args, **kwargs) 2023-03-08T07:07:40.1261283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T07:07:40.1261611Z return super().run(*args) 2023-03-08T07:07:40.1262090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T07:07:40.1262438Z self.env[node] = self.run_node(node) 2023-03-08T07:07:40.1262937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T07:07:40.1263265Z result = super().run_node(n) 2023-03-08T07:07:40.1263746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T07:07:40.1264131Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T07:07:40.1264670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T07:07:40.1265086Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T07:07:40.1265638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T07:07:40.1266002Z out = lowerings[target](*args, **kwargs) 2023-03-08T07:07:40.1266497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T07:07:40.1266835Z validate_ir(out) 2023-03-08T07:07:40.1267727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T07:07:40.1268079Z _check_tensorbox(node_or_nodes) 2023-03-08T07:07:40.1268598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T07:07:40.1268939Z assert isinstance( 2023-03-08T07:07:40.1269367Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T07:07:40.1270026Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T07:07:40.1270581Z target: aten._local_scalar_dense.default 2023-03-08T07:07:40.1270862Z args[0]: TensorBox(StorageBox( 2023-03-08T07:07:40.1271095Z Pointwise( 2023-03-08T07:07:40.1271345Z 'cpu', 2023-03-08T07:07:40.1271556Z torch.int64, 2023-03-08T07:07:40.1271766Z def inner_fn(index): 2023-03-08T07:07:40.1272034Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T07:07:40.1272327Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T07:07:40.1272594Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T07:07:40.1272844Z return tmp2 2023-03-08T07:07:40.1273053Z , 2023-03-08T07:07:40.1273234Z ranges=(), 2023-03-08T07:07:40.1273444Z origins={div} 2023-03-08T07:07:40.1273645Z ) 2023-03-08T07:07:40.1273815Z )) 2023-03-08T07:07:40.1273932Z 2023-03-08T07:07:40.1273937Z 2023-03-08T07:07:40.1274103Z You can suppress this exception and fall back to eager by setting: 2023-03-08T07:07:40.1274445Z torch._dynamo.config.suppress_errors = True 2023-03-08T07:07:40.1274627Z 2023-03-08T07:07:40.1274805Z TorchDynamo optimized model failed to run because of following error 2023-03-08T07:07:40.1313354Z FAIL 2023-03-08T07:08:25.7115313Z cuda train BartForCausalLM PASS 2023-03-08T07:09:51.5061938Z cuda train BartForConditionalGeneration PASS 2023-03-08T07:10:27.7344888Z cuda train BertForMaskedLM PASS 2023-03-08T07:11:01.6944645Z cuda train BertForQuestionAnswering PASS 2023-03-08T07:12:04.1071714Z cuda train BlenderbotForCausalLM PASS 2023-03-08T07:12:47.7673758Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T07:14:01.5957650Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T07:14:37.6339488Z cuda train CamemBert PASS 2023-03-08T07:15:21.1798414Z cuda train DebertaForMaskedLM PASS 2023-03-08T07:16:02.9160460Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T07:16:30.9342280Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T07:17:53.6248980Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T07:18:01.0081556Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T07:18:36.8543606Z cuda train DistilBertForMaskedLM PASS 2023-03-08T07:18:43.5463429Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T07:19:10.3306320Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T07:19:32.4583973Z cuda train DistillGPT2 PASS 2023-03-08T07:19:37.4345501Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T07:20:18.9524241Z cuda train ElectraForCausalLM PASS 2023-03-08T07:20:55.0535277Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T07:21:31.6626003Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T07:21:39.5914702Z Eager model failed to run 2023-03-08T07:21:39.5925631Z Traceback (most recent call last): 2023-03-08T07:21:39.5926274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1142, in validate_model 2023-03-08T07:21:39.5926682Z self.model_iter_fn(model, example_inputs) 2023-03-08T07:21:39.5927357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T07:21:39.5928444Z pred = mod(**cloned_inputs) 2023-03-08T07:21:39.5930540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5931177Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5932072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 759, in forward 2023-03-08T07:21:39.5932745Z outputs = self.fnet( 2023-03-08T07:21:39.5934351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5935496Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5936557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 600, in forward 2023-03-08T07:21:39.5936947Z encoder_outputs = self.encoder( 2023-03-08T07:21:39.5937455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5937869Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5938405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 304, in forward 2023-03-08T07:21:39.5938800Z layer_outputs = layer_module(hidden_states) 2023-03-08T07:21:39.5939629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5940058Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5940606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 263, in forward 2023-03-08T07:21:39.5941347Z self_fourier_outputs = self.fourier(hidden_states) 2023-03-08T07:21:39.5942310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5942659Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5943211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 216, in forward 2023-03-08T07:21:39.5943598Z self_outputs = self.self(hidden_states) 2023-03-08T07:21:39.5944092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T07:21:39.5944453Z return forward_call(*args, **kwargs) 2023-03-08T07:21:39.5944984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 195, in forward 2023-03-08T07:21:39.5945392Z outputs = self.fourier_transform(hidden_states).real 2023-03-08T07:21:39.5945831Z RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[512, 768] 2023-03-08T07:21:39.5946126Z 2023-03-08T07:21:39.5946299Z The above exception was the direct cause of the following exception: 2023-03-08T07:21:39.5946507Z 2023-03-08T07:21:39.5946622Z Traceback (most recent call last): 2023-03-08T07:21:39.5946960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2251, in run 2023-03-08T07:21:39.5947637Z ) = runner.load_model(device, model_name, batch_size=batch_size) 2023-03-08T07:21:39.5948092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 438, in load_model 2023-03-08T07:21:39.5948452Z self.validate_model(model, example_inputs) 2023-03-08T07:21:39.5948803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1144, in validate_model 2023-03-08T07:21:39.5949198Z raise NotImplementedError("Eager model failed to run") from e 2023-03-08T07:21:39.5949545Z NotImplementedError: Eager model failed to run 2023-03-08T07:21:39.5949733Z 2023-03-08T07:21:39.5949856Z WARNING:root:GoogleFnet failed to load 2023-03-08T07:22:15.8960688Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T07:22:55.0734558Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T07:23:20.6515628Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T07:24:45.0253620Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T07:25:31.7533162Z cuda train MBartForCausalLM PASS 2023-03-08T07:27:01.9192228Z cuda train MBartForConditionalGeneration PASS 2023-03-08T07:27:13.3140864Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T07:28:17.6225017Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T07:28:23.1525518Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T07:29:24.5656345Z cuda train MegatronBertForCausalLM PASS 2023-03-08T07:30:30.3596276Z cuda train MegatronBertForQuestionAnswering [2023-03-08 07:30:30,357] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.01052, (ref-fp64): 0.00057 and shape=torch.Size([1, 512]) 2023-03-08T07:30:30.3597206Z [2023-03-08 07:30:30,358] torch._dynamo.utils: [ERROR] Accuracy failed for key name start_logits 2023-03-08T07:30:30.3728339Z FAIL 2023-03-08T07:32:31.2024558Z cuda train MobileBertForMaskedLM PASS 2023-03-08T07:34:22.6543069Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T07:35:05.4636892Z cuda train OPTForCausalLM PASS 2023-03-08T07:35:36.4826071Z cuda train PLBartForCausalLM PASS 2023-03-08T07:36:34.1287241Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T07:36:48.5854759Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T07:37:29.2852922Z cuda train PegasusForCausalLM PASS 2023-03-08T07:37:53.5833690Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T07:39:14.3478906Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T07:39:19.8924548Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T07:39:50.4658589Z cuda train RobertaForCausalLM PASS 2023-03-08T07:40:24.6371868Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T07:40:30.5084646Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T07:41:01.1603335Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T07:41:41.4068790Z cuda train T5ForConditionalGeneration PASS 2023-03-08T07:42:22.0869502Z cuda train T5Small PASS 2023-03-08T07:43:08.3331913Z cuda train TrOCRForCausalLM PASS 2023-03-08T07:43:30.1658365Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T07:44:34.0083171Z cuda train XGLMForCausalLM PASS 2023-03-08T07:45:22.9805250Z cuda train XLNetLMHeadModel [2023-03-08 07:45:22,978] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T07:46:00.4675141Z PASS 2023-03-08T07:47:11.7431198Z cuda train YituTechConvBert PASS 2023-03-08T07:47:13.3745525Z accuracy pass_rate=91.11% 2023-03-08T07:47:13.3752791Z calls_captured gmean=414.53x mean=614.489x 2023-03-08T07:47:13.3762802Z unique_graphs gmean=3.30x mean=10.133x 2023-03-08T07:47:13.3770269Z graph_breaks gmean=9.52x mean=13.489x 2023-03-08T07:47:13.3777053Z unique_graph_breaks gmean=5.88x mean=6.600x 2023-03-08T07:47:16.1031524Z + test_single_dynamo_benchmark float32 huggingface '' --training --dtypes=float32 2023-03-08T07:47:16.1032730Z ++ pwd 2023-03-08T07:47:16.1037536Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2023-03-08T07:47:16.1038303Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2023-03-08T07:47:16.1054137Z + local name=float32 2023-03-08T07:47:16.1054504Z + shift 2023-03-08T07:47:16.1054890Z + local suite=huggingface 2023-03-08T07:47:16.1055276Z + shift 2023-03-08T07:47:16.1057355Z + local shard_id= 2023-03-08T07:47:16.1058163Z + shift 2023-03-08T07:47:16.1058474Z + partition_flags=() 2023-03-08T07:47:16.1058895Z + local partition_flags 2023-03-08T07:47:16.1059566Z + [[ -n 1 ]] 2023-03-08T07:47:16.1059923Z + [[ -n '' ]] 2023-03-08T07:47:16.1060179Z + [[ inductor_huggingface_perf == *perf* ]] 2023-03-08T07:47:16.1060458Z + MKL_THREADING_LAYER=GNU 2023-03-08T07:47:16.1061298Z + python benchmarks/dynamo/runner.py --suites=huggingface --base-sha=c88aa336aa0734f42b4d9db7f624d6cfd9b5065e --output-dir=/var/lib/jenkins/workspace/test/test-reports --no-graphs --no-update-archive --no-gh-comment --training --dtypes=float32 2023-03-08T07:47:20.4010071Z + rm -rf /var/lib/jenkins/workspace/test/test-reports 2023-03-08T07:47:20.4021759Z + mkdir /var/lib/jenkins/workspace/test/test-reports 2023-03-08T07:47:20.4039127Z + python benchmarks/dynamo/huggingface.py --performance --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/eager_huggingface_float32_training_cuda_performance.csv --training --backend=eager --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer 2023-03-08T07:47:33.2620453Z WARNING:__main__:Running smaller batch size=4 for AlbertForMaskedLM, orig batch_size=8 2023-03-08T07:48:02.8506901Z cuda train AlbertForMaskedLM 1.001x SAME 2023-03-08T07:48:12.2415853Z WARNING:__main__:Running smaller batch size=4 for AlbertForQuestionAnswering, orig batch_size=8 2023-03-08T07:48:41.7642280Z cuda train AlbertForQuestionAnswering 0.999x SAME 2023-03-08T07:48:50.5876582Z WARNING:__main__:Running smaller batch size=4 for AllenaiLongformerBase, orig batch_size=8 2023-03-08T07:49:19.2391038Z cuda train AllenaiLongformerBase 0.977x p=0.00 2023-03-08T07:49:30.7679523Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T07:49:43.9742108Z cuda train BartForCausalLM 0.998x p=0.00 2023-03-08T07:50:03.0777668Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T07:50:22.4883270Z cuda train BartForConditionalGeneration 0.997x p=0.00 2023-03-08T07:50:30.4340193Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T07:50:44.3249930Z cuda train BertForMaskedLM 0.996x p=0.00 2023-03-08T07:50:51.9203334Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T07:51:04.1186637Z cuda train BertForQuestionAnswering 0.995x p=0.00 2023-03-08T07:52:05.3778342Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T07:52:20.1030142Z cuda train BlenderbotForCausalLM 0.973x p=0.00 2023-03-08T07:52:26.8299031Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T07:52:34.7232256Z cuda train BlenderbotSmallForCausalLM 0.997x p=0.05 2023-03-08T07:52:43.3115779Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T07:52:57.9030483Z cuda train BlenderbotSmallForConditionalGeneration 0.997x p=0.00 2023-03-08T07:53:05.6489679Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T07:53:19.5748474Z cuda train CamemBert 0.995x p=0.00 2023-03-08T07:53:27.7978205Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T07:53:39.1747395Z cuda train DebertaForMaskedLM 0.838x p=0.00 2023-03-08T07:53:46.8824319Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T07:53:59.5454412Z cuda train DebertaForQuestionAnswering 0.974x p=0.00 2023-03-08T07:54:26.9089166Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T07:54:36.3461046Z cuda train DebertaV2ForMaskedLM [2023-03-08 07:54:36,344] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T07:54:36.3461855Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T07:54:36.3462275Z reasons: ___check_obj_id(self, 140182664538448) 2023-03-08T07:54:36.3465712Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T07:54:47.0837956Z 0.775x p=0.00 2023-03-08T07:55:11.6055060Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T07:55:21.2274436Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 07:55:21,226] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T07:55:21.2275245Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T07:55:21.2275648Z reasons: ___check_obj_id(self, 139788048773280) 2023-03-08T07:55:21.2278986Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T07:55:32.9331676Z 0.808x p=0.00 2023-03-08T07:55:39.8174844Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T07:55:39.8175363Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T07:55:50.5127921Z cuda train DistilBertForMaskedLM 0.998x p=0.00 2023-03-08T07:55:56.8080182Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T07:55:56.8081158Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T07:56:09.0880054Z cuda train DistilBertForQuestionAnswering 0.998x p=0.00 2023-03-08T07:56:16.5924294Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T07:56:27.3175523Z cuda train DistillGPT2 0.997x p=0.00 2023-03-08T07:56:31.9571015Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T07:56:32.5381051Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T07:56:44.4358073Z cuda train ElectraForCausalLM 0.995x p=0.00 2023-03-08T07:56:49.5833624Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T07:57:02.7923379Z cuda train ElectraForQuestionAnswering 0.995x p=0.00 2023-03-08T07:57:10.9555522Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T07:57:22.2212436Z cuda train GPT2ForSequenceClassification 0.995x p=0.00 2023-03-08T07:57:29.3100503Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T07:57:39.1835632Z cuda train GoogleFnet 0.997x p=0.00 2023-03-08T07:57:46.9197215Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T07:58:01.2756581Z cuda train LayoutLMForMaskedLM 0.996x p=0.00 2023-03-08T07:58:08.6831482Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T07:58:21.3510647Z cuda train LayoutLMForSequenceClassification 0.994x p=0.00 2023-03-08T07:58:46.4631450Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T07:58:46.4632052Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T07:59:03.3800300Z cuda train M2M100ForConditionalGeneration 1.032x p=0.00 2023-03-08T07:59:14.8655051Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T07:59:28.0417603Z cuda train MBartForCausalLM 0.998x p=0.00 2023-03-08T07:59:47.1060280Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T08:00:06.4559039Z cuda train MBartForConditionalGeneration 0.997x p=0.02 2023-03-08T08:00:16.3907413Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T08:00:16.3907939Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T08:00:29.1819129Z cuda train MT5ForConditionalGeneration 1.012x p=0.00 2023-03-08T08:00:33.9698948Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T08:00:42.0719261Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T08:00:57.7058356Z cuda train MegatronBertForCausalLM 0.991x p=0.00 2023-03-08T08:01:10.0288365Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T08:01:30.7047617Z cuda train MegatronBertForQuestionAnswering 0.993x p=0.00 2023-03-08T08:01:36.5965913Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T08:01:58.5664744Z cuda train MobileBertForMaskedLM 0.991x p=0.00 2023-03-08T08:02:04.1403017Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T08:02:27.7043813Z cuda train MobileBertForQuestionAnswering 0.990x p=0.01 2023-03-08T08:02:36.8811896Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T08:02:53.2421030Z cuda train OPTForCausalLM 0.998x p=0.00 2023-03-08T08:03:00.8412543Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T08:03:12.1846830Z cuda train PLBartForCausalLM 0.998x SAME 2023-03-08T08:03:22.8848772Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T08:03:38.0723302Z cuda train PLBartForConditionalGeneration 0.997x p=0.00 2023-03-08T08:03:51.7260363Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T08:03:51.7261042Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:04:02.0230787Z cuda train PegasusForCausalLM 0.996x p=0.00 2023-03-08T08:04:25.8778344Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T08:04:25.8778999Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T08:04:43.4711992Z cuda train PegasusForConditionalGeneration 0.995x p=0.00 2023-03-08T08:04:48.2005696Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T08:04:51.2116977Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T08:05:05.6254365Z cuda train RobertaForCausalLM 0.996x p=0.00 2023-03-08T08:05:12.8910809Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T08:05:25.1001665Z cuda train RobertaForQuestionAnswering 0.994x SAME 2023-03-08T08:05:30.3478090Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T08:05:30.3478604Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:05:37.4769260Z cuda train Speech2Text2ForCausalLM 0.995x p=0.00 2023-03-08T08:05:43.7461987Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T08:05:56.2462152Z cuda train T5ForConditionalGeneration 0.993x p=0.00 2023-03-08T08:06:02.5731587Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T08:06:15.0776303Z cuda train T5Small 0.995x p=0.01 2023-03-08T08:06:26.3273536Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T08:06:42.6058949Z cuda train TrOCRForCausalLM 0.998x p=0.01 2023-03-08T08:07:03.8674158Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T08:07:03.8674886Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:07:16.8843343Z cuda train XGLMForCausalLM 0.965x p=0.00 2023-03-08T08:07:29.5754309Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T08:08:05.1071469Z cuda train XLNetLMHeadModel 0.999x p=0.04 2023-03-08T08:08:12.7681193Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T08:08:29.5496756Z cuda train YituTechConvBert 0.996x p=0.00 2023-03-08T08:08:30.2594943Z speedup gmean=1.00x mean=1.001x 2023-03-08T08:08:30.2600637Z abs_latency gmean=128.71x mean=140.616x 2023-03-08T08:08:30.2603113Z compilation_latency mean=5.516 seconds 2023-03-08T08:08:30.2603442Z compression_ratio mean=0.988x 2023-03-08T08:08:30.7447167Z + python benchmarks/dynamo/huggingface.py --performance --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/aot_eager_huggingface_float32_training_cuda_performance.csv --training --backend=aot_eager --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer 2023-03-08T08:08:43.4815754Z WARNING:__main__:Running smaller batch size=4 for AlbertForMaskedLM, orig batch_size=8 2023-03-08T08:09:18.6523553Z cuda train AlbertForMaskedLM 0.998x p=0.00 2023-03-08T08:09:28.0565902Z WARNING:__main__:Running smaller batch size=4 for AlbertForQuestionAnswering, orig batch_size=8 2023-03-08T08:10:02.9850431Z cuda train AlbertForQuestionAnswering 1.001x p=0.03 2023-03-08T08:10:11.8351397Z WARNING:__main__:Running smaller batch size=4 for AllenaiLongformerBase, orig batch_size=8 2023-03-08T08:11:11.8432840Z cuda train AllenaiLongformerBase 0.590x p=0.00 2023-03-08T08:11:23.6043320Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T08:11:42.2554341Z cuda train BartForCausalLM 0.965x p=0.00 2023-03-08T08:12:01.4353429Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T08:12:34.5197121Z cuda train BartForConditionalGeneration 0.981x p=0.00 2023-03-08T08:12:42.4154905Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T08:13:02.0210285Z cuda train BertForMaskedLM 0.961x p=0.00 2023-03-08T08:13:09.3821561Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T08:13:27.2012352Z cuda train BertForQuestionAnswering 0.980x p=0.00 2023-03-08T08:14:28.4076920Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T08:14:53.9567056Z cuda train BlenderbotForCausalLM 0.903x p=0.00 2023-03-08T08:15:00.8170358Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T08:15:12.6089740Z cuda train BlenderbotSmallForCausalLM 0.914x p=0.00 2023-03-08T08:15:21.2272398Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T08:15:45.2082973Z cuda train BlenderbotSmallForConditionalGeneration 0.935x p=0.01 2023-03-08T08:15:53.0535092Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T08:16:12.8663112Z cuda train CamemBert 0.960x p=0.00 2023-03-08T08:16:21.1416026Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T08:16:39.6185845Z cuda train DebertaForMaskedLM 0.705x p=0.00 2023-03-08T08:16:47.4431724Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T08:17:07.2412160Z cuda train DebertaForQuestionAnswering 0.845x p=0.00 2023-03-08T08:17:34.5535240Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T08:17:53.5078672Z cuda train DebertaV2ForMaskedLM [2023-03-08 08:17:53,506] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T08:17:53.5079727Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T08:17:53.5080142Z reasons: ___check_obj_id(self, 140567316476208) 2023-03-08T08:17:53.5080555Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T08:18:07.0754926Z 0.593x p=0.00 2023-03-08T08:18:31.8080633Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T08:18:51.0111700Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 08:18:51,010] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T08:18:51.0112489Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T08:18:51.0112984Z reasons: ___check_obj_id(self, 140324578357456) 2023-03-08T08:18:51.0116997Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T08:19:05.3918725Z 0.677x p=0.00 2023-03-08T08:19:12.5473488Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T08:19:12.5474011Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T08:19:26.5084878Z cuda train DistilBertForMaskedLM 0.945x p=0.00 2023-03-08T08:19:32.9234185Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T08:19:32.9234898Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T08:19:48.2097777Z cuda train DistilBertForQuestionAnswering 0.990x p=0.00 2023-03-08T08:19:55.7210861Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T08:20:09.5261011Z cuda train DistillGPT2 0.944x p=0.00 2023-03-08T08:20:14.1281547Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T08:20:14.7198166Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T08:20:32.6068835Z cuda train ElectraForCausalLM 0.930x p=0.00 2023-03-08T08:20:37.8754182Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T08:20:56.8664226Z cuda train ElectraForQuestionAnswering 0.983x p=0.00 2023-03-08T08:21:05.0810279Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T08:21:21.7833435Z cuda train GPT2ForSequenceClassification 0.968x p=0.00 2023-03-08T08:21:28.9192213Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T08:21:41.6377693Z cuda train GoogleFnet 0.967x p=0.00 2023-03-08T08:21:49.4129044Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T08:22:09.7540184Z cuda train LayoutLMForMaskedLM 0.961x p=0.00 2023-03-08T08:22:17.1815639Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T08:22:35.8635509Z cuda train LayoutLMForSequenceClassification 0.976x p=0.00 2023-03-08T08:23:01.1883880Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T08:23:01.1884409Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T08:23:31.9696086Z cuda train M2M100ForConditionalGeneration 0.954x p=0.00 2023-03-08T08:23:43.5459368Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T08:24:02.2364154Z cuda train MBartForCausalLM 0.964x p=0.00 2023-03-08T08:24:21.4475134Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T08:24:54.6146098Z cuda train MBartForConditionalGeneration 0.980x p=0.00 2023-03-08T08:25:04.7820642Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T08:25:04.7821220Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T08:25:29.3232776Z cuda train MT5ForConditionalGeneration 0.879x p=0.00 2023-03-08T08:25:34.2710568Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T08:25:42.3988425Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T08:26:09.1211182Z cuda train MegatronBertForCausalLM 0.978x p=0.00 2023-03-08T08:26:21.5717323Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T08:26:53.4380468Z cuda train MegatronBertForQuestionAnswering 0.974x p=0.00 2023-03-08T08:26:59.4101007Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T08:27:43.1737493Z cuda train MobileBertForMaskedLM 0.874x p=0.00 2023-03-08T08:27:49.1145372Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T08:28:34.4974832Z cuda train MobileBertForQuestionAnswering 0.924x p=0.00 2023-03-08T08:28:43.9650110Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T08:29:06.0702279Z cuda train OPTForCausalLM 0.933x p=0.00 2023-03-08T08:29:13.6872301Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T08:29:28.0546177Z cuda train PLBartForCausalLM 0.948x p=0.00 2023-03-08T08:29:38.7844098Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T08:30:01.4552897Z cuda train PLBartForConditionalGeneration 0.958x p=0.00 2023-03-08T08:30:15.3448113Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T08:30:15.3449701Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:30:31.1535350Z cuda train PegasusForCausalLM 0.951x p=0.00 2023-03-08T08:30:55.2332260Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T08:30:55.2332839Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T08:31:26.7003794Z cuda train PegasusForConditionalGeneration 0.976x p=0.00 2023-03-08T08:31:31.6640475Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T08:31:34.6740622Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T08:31:55.1384524Z cuda train RobertaForCausalLM 0.959x p=0.00 2023-03-08T08:32:02.6715800Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T08:32:20.8203915Z cuda train RobertaForQuestionAnswering 0.977x p=0.00 2023-03-08T08:32:26.1475592Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T08:32:26.1476114Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:32:36.2704392Z cuda train Speech2Text2ForCausalLM 0.921x p=0.00 2023-03-08T08:32:42.6515295Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T08:33:03.7929409Z cuda train T5ForConditionalGeneration 0.818x p=0.00 2023-03-08T08:33:10.2263049Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T08:33:31.3179606Z cuda train T5Small 0.818x p=0.00 2023-03-08T08:33:42.7590150Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T08:34:04.6988037Z cuda train TrOCRForCausalLM 0.957x p=0.00 2023-03-08T08:34:25.9700822Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T08:34:25.9701319Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T08:34:51.0591382Z cuda train XGLMForCausalLM 0.859x p=0.00 2023-03-08T08:35:03.9996113Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T08:35:59.6415319Z cuda train XLNetLMHeadModel 0.957x p=0.00 2023-03-08T08:36:07.6852821Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T08:36:33.7089826Z cuda train YituTechConvBert 0.958x p=0.00 2023-03-08T08:36:34.6724365Z speedup gmean=1.00x mean=1.000x 2023-03-08T08:36:34.6730443Z abs_latency gmean=138.65x mean=152.024x 2023-03-08T08:36:34.6731315Z compilation_latency mean=14.061 seconds 2023-03-08T08:36:34.6733008Z compression_ratio mean=0.918x 2023-03-08T08:36:35.2143873Z + python benchmarks/dynamo/huggingface.py --performance --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_huggingface_float32_training_cuda_performance.csv --training --inductor --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer --cold-start-latency 2023-03-08T08:36:48.2144870Z WARNING:__main__:Running smaller batch size=4 for AlbertForMaskedLM, orig batch_size=8 2023-03-08T08:37:18.2656053Z cuda train AlbertForMaskedLM [2023-03-08 08:37:18,264] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:37:43.3921240Z 1.256x p=0.00 2023-03-08T08:37:53.4211201Z WARNING:__main__:Running smaller batch size=4 for AlbertForQuestionAnswering, orig batch_size=8 2023-03-08T08:38:21.8443538Z cuda train AlbertForQuestionAnswering [2023-03-08 08:38:21,843] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:38:45.0488714Z 1.260x p=0.00 2023-03-08T08:38:54.5561148Z WARNING:__main__:Running smaller batch size=4 for AllenaiLongformerBase, orig batch_size=8 2023-03-08T08:38:58.6353546Z cuda train AllenaiLongformerBase [2023-03-08 08:38:58,633] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:04.3090916Z [2023-03-08 08:39:04,308] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T08:39:06.4596806Z [2023-03-08 08:39:06,458] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T08:39:10.1276405Z [2023-03-08 08:39:10,126] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T08:39:10.1669039Z ERROR:common:Backend dynamo failed in warmup() 2023-03-08T08:39:10.1671253Z Traceback (most recent call last): 2023-03-08T08:39:10.1671663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1372, in warmup 2023-03-08T08:39:10.1671979Z fn(model, example_inputs) 2023-03-08T08:39:10.1672722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T08:39:10.1675527Z return fn(*args, **kwargs) 2023-03-08T08:39:10.1675959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T08:39:10.1676338Z cloned_inputs = clone_inputs(inputs) 2023-03-08T08:39:10.1676743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T08:39:10.1678017Z self.optimizer_zero_grad(mod) 2023-03-08T08:39:10.1679210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T08:39:10.1679636Z pred = mod(**cloned_inputs) 2023-03-08T08:39:10.1680271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1680629Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1681205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T08:39:10.1681593Z outputs = self.longformer( 2023-03-08T08:39:10.1682095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1682443Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1683218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T08:39:10.1683864Z encoder_outputs = self.encoder( 2023-03-08T08:39:10.1684644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1685637Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1686218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T08:39:10.1686653Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T08:39:10.1687272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T08:39:10.1687718Z layer_outputs = layer_module( 2023-03-08T08:39:10.1688220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1688583Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1689133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T08:39:10.1689532Z self_attn_outputs = self.attention( 2023-03-08T08:39:10.1690040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1690400Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1690939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T08:39:10.1691317Z self_outputs = self.self( 2023-03-08T08:39:10.1691810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T08:39:10.1692167Z return forward_call(*args, **kwargs) 2023-03-08T08:39:10.1692705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T08:39:10.1693124Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T08:39:10.1693731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T08:39:10.1694160Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T08:39:10.1694792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T08:39:10.1695296Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T08:39:10.1695862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T08:39:10.1696222Z return callback(frame, cache_size, hooks) 2023-03-08T08:39:10.1696748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T08:39:10.1697243Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T08:39:10.1697813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T08:39:10.1698155Z return fn(*args, **kwargs) 2023-03-08T08:39:10.1698676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T08:39:10.1699029Z return _compile( 2023-03-08T08:39:10.1699491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T08:39:10.1699831Z r = func(*args, **kwargs) 2023-03-08T08:39:10.1700321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T08:39:10.1700692Z out_code = transform_code_object(code, transform) 2023-03-08T08:39:10.1701274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T08:39:10.1701766Z transformations(instructions, code_options) 2023-03-08T08:39:10.1702294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T08:39:10.1702618Z tracer.run() 2023-03-08T08:39:10.1703101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T08:39:10.1703434Z super().run() 2023-03-08T08:39:10.1703896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T08:39:10.1704228Z and self.step() 2023-03-08T08:39:10.1704710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T08:39:10.1705073Z getattr(self, inst.opname)(inst) 2023-03-08T08:39:10.1705580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T08:39:10.1705981Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T08:39:10.1706536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T08:39:10.1706948Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T08:39:10.1707780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T08:39:10.1708184Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T08:39:10.1708696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T08:39:10.1709027Z r = func(*args, **kwargs) 2023-03-08T08:39:10.1709536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T08:39:10.1709978Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T08:39:10.1710540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T08:39:10.1710954Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T08:39:10.1711505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T08:39:10.1711893Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T08:39:10.1712406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T08:39:10.1712770Z return compile_fx(*args, **kwargs) 2023-03-08T08:39:10.1713285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T08:39:10.1713622Z return aot_autograd( 2023-03-08T08:39:10.1714207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T08:39:10.1714627Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T08:39:10.1715201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T08:39:10.1715592Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T08:39:10.1716103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T08:39:10.1716445Z r = func(*args, **kwargs) 2023-03-08T08:39:10.1716974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T08:39:10.1717470Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T08:39:10.1718040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T08:39:10.1718457Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T08:39:10.1719081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T08:39:10.1719520Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T08:39:10.1720067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T08:39:10.1720402Z r = func(*args, **kwargs) 2023-03-08T08:39:10.1720880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T08:39:10.1721230Z return inner_compile( 2023-03-08T08:39:10.1721721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T08:39:10.1722095Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T08:39:10.1722603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T08:39:10.1722949Z return fn(*args, **kwargs) 2023-03-08T08:39:10.1723250Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T08:39:10.1723557Z return func(*args, **kwds) 2023-03-08T08:39:10.1724064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T08:39:10.1724423Z graph.run(*example_inputs) 2023-03-08T08:39:10.1724895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T08:39:10.1725230Z r = func(*args, **kwargs) 2023-03-08T08:39:10.1725699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T08:39:10.1726024Z return super().run(*args) 2023-03-08T08:39:10.1726493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T08:39:10.1726843Z self.env[node] = self.run_node(node) 2023-03-08T08:39:10.1727373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T08:39:10.1727707Z result = super().run_node(n) 2023-03-08T08:39:10.1728189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T08:39:10.1728569Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T08:39:10.1729078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T08:39:10.1729491Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T08:39:10.1730045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T08:39:10.1730407Z out = lowerings[target](*args, **kwargs) 2023-03-08T08:39:10.1730956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T08:39:10.1731300Z validate_ir(out) 2023-03-08T08:39:10.1731773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T08:39:10.1732111Z _check_tensorbox(node_or_nodes) 2023-03-08T08:39:10.1732604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T08:39:10.1732942Z assert isinstance( 2023-03-08T08:39:10.1733342Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T08:39:10.1734010Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T08:39:10.1734450Z target: aten._local_scalar_dense.default 2023-03-08T08:39:10.1734726Z args[0]: TensorBox(StorageBox( 2023-03-08T08:39:10.1734948Z Pointwise( 2023-03-08T08:39:10.1735188Z 'cpu', 2023-03-08T08:39:10.1735401Z torch.int64, 2023-03-08T08:39:10.1735669Z def inner_fn(index): 2023-03-08T08:39:10.1735931Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T08:39:10.1736220Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T08:39:10.1736488Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T08:39:10.1736735Z return tmp2 2023-03-08T08:39:10.1736943Z , 2023-03-08T08:39:10.1737129Z ranges=(), 2023-03-08T08:39:10.1737380Z origins={div} 2023-03-08T08:39:10.1737584Z ) 2023-03-08T08:39:10.1737753Z )) 2023-03-08T08:39:10.1737870Z 2023-03-08T08:39:10.1737875Z 2023-03-08T08:39:10.1738048Z You can suppress this exception and fall back to eager by setting: 2023-03-08T08:39:10.1738391Z torch._dynamo.config.suppress_errors = True 2023-03-08T08:39:10.1738575Z 2023-03-08T08:39:11.1262258Z ERROR 2023-03-08T08:39:21.8405152Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T08:39:24.6407014Z cuda train BartForCausalLM [2023-03-08 08:39:24,639] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:28.9459541Z [2023-03-08 08:39:28,945] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:32.1239805Z [2023-03-08 08:39:32,122] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:33.2367759Z [2023-03-08 08:39:33,235] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:34.3564340Z [2023-03-08 08:39:34,355] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:35.4680796Z [2023-03-08 08:39:35,467] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:36.5755491Z [2023-03-08 08:39:36,574] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:37.6827018Z [2023-03-08 08:39:37,681] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:38.8073342Z [2023-03-08 08:39:38,806] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:39.9299013Z [2023-03-08 08:39:39,929] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:41.0539595Z [2023-03-08 08:39:41,053] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:42.1762681Z [2023-03-08 08:39:42,175] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:39:43.3126994Z [2023-03-08 08:39:43,312] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:40:04.0601054Z 1.141x p=0.00 2023-03-08T08:40:23.7589997Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T08:40:50.6633551Z cuda train BartForConditionalGeneration [2023-03-08 08:40:50,661] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:41:36.2277633Z 1.091x p=0.00 2023-03-08T08:41:45.2344938Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T08:41:57.9667742Z cuda train BertForMaskedLM [2023-03-08 08:41:57,965] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:42:13.9889651Z [2023-03-08 08:42:13,987] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:42:24.9029569Z 1.193x p=0.00 2023-03-08T08:42:33.0748337Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T08:42:46.0164813Z cuda train BertForQuestionAnswering [2023-03-08 08:42:46,015] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:43:01.9714473Z [2023-03-08 08:43:01,970] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:43:10.9460998Z 1.264x p=0.00 2023-03-08T08:44:13.2888477Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T08:44:16.0431848Z cuda train BlenderbotForCausalLM [2023-03-08 08:44:16,042] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:17.7085930Z [2023-03-08 08:44:17,707] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:22.6911867Z [2023-03-08 08:44:22,690] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:23.8113087Z [2023-03-08 08:44:23,810] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:24.9286805Z [2023-03-08 08:44:24,927] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:26.0388282Z [2023-03-08 08:44:26,038] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:27.1563772Z [2023-03-08 08:44:27,155] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:28.2734935Z [2023-03-08 08:44:28,272] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:29.3844590Z [2023-03-08 08:44:29,383] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:30.4983424Z [2023-03-08 08:44:30,497] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:31.6153342Z [2023-03-08 08:44:31,614] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:32.7369314Z [2023-03-08 08:44:32,736] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:33.8515666Z [2023-03-08 08:44:33,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:34.9640075Z [2023-03-08 08:44:34,963] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:36.0921218Z [2023-03-08 08:44:36,091] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:37.2319366Z [2023-03-08 08:44:37,231] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:38.3609701Z [2023-03-08 08:44:38,360] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:39.4921632Z [2023-03-08 08:44:39,491] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:40.6207994Z [2023-03-08 08:44:40,620] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:41.7584713Z [2023-03-08 08:44:41,757] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:42.8925858Z [2023-03-08 08:44:42,891] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:44.0349528Z [2023-03-08 08:44:44,034] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:45.1886007Z [2023-03-08 08:44:45,187] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:46.3275358Z [2023-03-08 08:44:46,326] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:44:47.4690707Z [2023-03-08 08:44:47,468] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:11.8873361Z ERROR:common:Backend dynamo failed in warmup() 2023-03-08T08:45:11.8873710Z Traceback (most recent call last): 2023-03-08T08:45:11.8874055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1372, in warmup 2023-03-08T08:45:11.8874398Z fn(model, example_inputs) 2023-03-08T08:45:11.8875627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T08:45:11.8876382Z return fn(*args, **kwargs) 2023-03-08T08:45:11.8876754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T08:45:11.8877136Z cloned_inputs = clone_inputs(inputs) 2023-03-08T08:45:11.8877622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T08:45:11.8877992Z self.optimizer_zero_grad(mod) 2023-03-08T08:45:11.8878385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 493, in 2023-03-08T08:45:11.8878785Z self.grad_scaler.scale(loss).backward() 2023-03-08T08:45:11.8881468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward 2023-03-08T08:45:11.8881904Z torch.autograd.backward( 2023-03-08T08:45:11.8882445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 204, in backward 2023-03-08T08:45:11.8882918Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-03-08T08:45:11.8883676Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 39.45 GiB total capacity; 36.83 GiB already allocated; 79.25 MiB free; 38.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T08:45:13.0136091Z ERROR 2023-03-08T08:45:19.0494323Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T08:45:21.5164287Z cuda train BlenderbotSmallForCausalLM [2023-03-08 08:45:21,515] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:23.2426581Z [2023-03-08 08:45:23,241] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:26.7861751Z [2023-03-08 08:45:26,785] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:27.8680459Z [2023-03-08 08:45:27,867] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:28.9497992Z [2023-03-08 08:45:28,949] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:30.0387542Z [2023-03-08 08:45:30,038] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:31.1276781Z [2023-03-08 08:45:31,126] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:32.2221362Z [2023-03-08 08:45:32,221] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:33.3123854Z [2023-03-08 08:45:33,311] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:45:48.9669229Z 0.999x SAME 2023-03-08T08:45:57.8315359Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T08:46:16.2653394Z cuda train BlenderbotSmallForConditionalGeneration [2023-03-08 08:46:16,264] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:46:50.8225850Z 1.071x p=0.00 2023-03-08T08:46:59.5867878Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T08:47:12.5177277Z cuda train CamemBert [2023-03-08 08:47:12,516] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:47:27.6753723Z [2023-03-08 08:47:27,674] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:47:37.4023482Z 1.192x p=0.00 2023-03-08T08:47:46.1754936Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T08:47:49.8070444Z cuda train DebertaForMaskedLM [2023-03-08 08:47:49,806] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:47:56.8257109Z [2023-03-08 08:47:56,824] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T08:47:56.8930008Z [2023-03-08 08:47:56,892] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:48:52.2117952Z 0.925x p=0.00 2023-03-08T08:49:00.4329385Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T08:49:03.9814903Z cuda train DebertaForQuestionAnswering [2023-03-08 08:49:03,980] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:49:09.0455744Z [2023-03-08 08:49:09,044] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T08:49:09.1112556Z [2023-03-08 08:49:09,110] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:50:06.9309888Z 0.990x SAME 2023-03-08T08:50:34.7895576Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T08:50:40.8542508Z cuda train DebertaV2ForMaskedLM [2023-03-08 08:50:40,853] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:50:43.2843010Z [2023-03-08 08:50:43,283] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T08:50:43.3459050Z [2023-03-08 08:50:43,345] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:51:29.8283053Z [2023-03-08 08:51:29,826] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T08:51:29.8283830Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T08:51:29.8284249Z reasons: ___check_obj_id(self, 139669485038192) 2023-03-08T08:51:29.8284838Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T08:52:31.0964901Z 0.798x p=0.00 2023-03-08T08:52:56.2665219Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T08:52:59.9718719Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 08:52:59,971] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:53:02.5124015Z [2023-03-08 08:53:02,511] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T08:53:02.5748027Z [2023-03-08 08:53:02,574] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:53:50.0146368Z [2023-03-08 08:53:50,013] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T08:53:50.0147611Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T08:53:50.0148139Z reasons: ___check_obj_id(self, 140049259294880) 2023-03-08T08:53:50.0148742Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T08:54:56.5549483Z 0.798x p=0.00 2023-03-08T08:55:04.1012387Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T08:55:04.1012908Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T08:55:11.1691901Z cuda train DistilBertForMaskedLM [2023-03-08 08:55:11,168] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:55:39.1310732Z 1.030x p=0.00 2023-03-08T08:55:45.9632941Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T08:55:45.9633531Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T08:55:53.2830889Z cuda train DistilBertForQuestionAnswering [2023-03-08 08:55:53,281] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:56:23.6983440Z 1.141x p=0.00 2023-03-08T08:56:31.6527681Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T08:56:38.8014480Z cuda train DistillGPT2 [2023-03-08 08:56:38,800] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:56:59.0714897Z 1.374x p=0.00 2023-03-08T08:57:04.1669009Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T08:57:04.7544974Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T08:57:17.0480810Z cuda train ElectraForCausalLM [2023-03-08 08:57:17,046] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:57:48.5011972Z [2023-03-08 08:57:48,499] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:57:57.7998082Z 1.390x p=0.00 2023-03-08T08:58:03.7041129Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T08:58:16.4372125Z cuda train ElectraForQuestionAnswering [2023-03-08 08:58:16,436] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:58:36.7781672Z [2023-03-08 08:58:36,777] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T08:58:46.4574391Z 1.423x p=0.00 2023-03-08T08:58:55.3367298Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T08:59:07.3926474Z cuda train GPT2ForSequenceClassification [2023-03-08 08:59:07,391] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:30.2740035Z 1.640x p=0.00 2023-03-08T08:59:37.8739596Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T08:59:40.2112323Z cuda train GoogleFnet [2023-03-08 08:59:40,210] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:43.9646624Z [2023-03-08 08:59:43,963] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:45.2908182Z [2023-03-08 08:59:45,290] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:46.0763038Z [2023-03-08 08:59:46,075] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:46.8681097Z [2023-03-08 08:59:46,867] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:47.6611280Z [2023-03-08 08:59:47,660] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:48.4701007Z [2023-03-08 08:59:48,469] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:49.2751998Z [2023-03-08 08:59:49,274] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:50.0937602Z [2023-03-08 08:59:50,093] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:50.9138960Z [2023-03-08 08:59:50,912] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:51.7274650Z [2023-03-08 08:59:51,726] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:52.5604831Z [2023-03-08 08:59:52,559] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T08:59:53.3898619Z [2023-03-08 08:59:53,388] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:00:04.6977310Z [2023-03-08 09:00:04,696] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T09:00:12.1059004Z 1.429x p=0.00 2023-03-08T09:00:20.1908590Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T09:00:33.6032212Z cuda train LayoutLMForMaskedLM [2023-03-08 09:00:33,602] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:01:01.1861192Z 1.176x p=0.00 2023-03-08T09:01:09.3348158Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T09:01:22.4000996Z cuda train LayoutLMForSequenceClassification [2023-03-08 09:01:22,399] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:01:53.6562592Z 1.282x p=0.00 2023-03-08T09:02:19.5125980Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T09:02:19.5127229Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T09:02:31.5116379Z cuda train M2M100ForConditionalGeneration [2023-03-08 09:02:31,510] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:02:47.8073488Z [2023-03-08 09:02:47,806] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:02:49.4440977Z [2023-03-08 09:02:49,443] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:02:55.4730865Z [2023-03-08 09:02:55,472] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:02:57.3071433Z [2023-03-08 09:02:57,306] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:02:59.0913526Z [2023-03-08 09:02:59,090] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:00.8839686Z [2023-03-08 09:03:00,883] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:02.6931637Z [2023-03-08 09:03:02,692] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:04.4838382Z [2023-03-08 09:03:04,483] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:06.2803850Z [2023-03-08 09:03:06,279] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:08.0563684Z [2023-03-08 09:03:08,055] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:09.8516448Z [2023-03-08 09:03:09,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:11.6499451Z [2023-03-08 09:03:11,649] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:03:13.4538065Z [2023-03-08 09:03:13,452] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:00.7485108Z 1.077x p=0.00 2023-03-08T09:04:12.9774131Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T09:04:15.7843706Z cuda train MBartForCausalLM [2023-03-08 09:04:15,783] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:20.0558092Z [2023-03-08 09:04:20,054] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:23.2215707Z [2023-03-08 09:04:23,220] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:24.3167660Z [2023-03-08 09:04:24,315] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:25.4286139Z [2023-03-08 09:04:25,428] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:26.5178099Z [2023-03-08 09:04:26,517] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:27.6071165Z [2023-03-08 09:04:27,606] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:28.7017612Z [2023-03-08 09:04:28,701] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:29.7879680Z [2023-03-08 09:04:29,787] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:30.8796296Z [2023-03-08 09:04:30,878] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:31.9823892Z [2023-03-08 09:04:31,981] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:33.0866602Z [2023-03-08 09:04:33,086] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:34.1920702Z [2023-03-08 09:04:34,191] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:04:55.0799036Z 1.140x p=0.00 2023-03-08T09:05:14.5186670Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T09:05:41.1701314Z cuda train MBartForConditionalGeneration [2023-03-08 09:05:41,168] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:06:29.7236912Z 1.089x p=0.00 2023-03-08T09:06:40.8795228Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T09:06:40.8795757Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T09:07:03.3096325Z cuda train MT5ForConditionalGeneration [2023-03-08 09:07:03,308] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:08:50.8040317Z 1.410x p=0.00 2023-03-08T09:08:56.4965576Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T09:09:04.5814694Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T09:09:27.6801973Z cuda train MegatronBertForCausalLM [2023-03-08 09:09:27,678] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:10:07.0821224Z 1.136x p=0.00 2023-03-08T09:10:20.3928693Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T09:10:43.7771366Z cuda train MegatronBertForQuestionAnswering [2023-03-08 09:10:43,775] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:11:26.5036304Z 1.197x p=0.00 2023-03-08T09:11:33.4157811Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T09:12:14.4964808Z cuda train MobileBertForMaskedLM [2023-03-08 09:12:14,494] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:13:34.2605039Z 1.403x p=0.00 2023-03-08T09:13:41.8323942Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T09:14:22.9293283Z cuda train MobileBertForQuestionAnswering [2023-03-08 09:14:22,927] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:15:39.7530289Z 1.344x p=0.00 2023-03-08T09:15:50.8408724Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T09:15:55.3486440Z cuda train OPTForCausalLM [2023-03-08 09:15:55,347] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:15:58.9319775Z [2023-03-08 09:15:58,931] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:00.0345920Z [2023-03-08 09:16:00,033] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:01.1301098Z [2023-03-08 09:16:01,129] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:02.2372446Z [2023-03-08 09:16:02,236] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:03.3414224Z [2023-03-08 09:16:03,340] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:04.4356570Z [2023-03-08 09:16:04,435] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:05.5171021Z [2023-03-08 09:16:05,516] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:06.6103262Z [2023-03-08 09:16:06,609] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:07.7128191Z [2023-03-08 09:16:07,712] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:08.8125623Z [2023-03-08 09:16:08,811] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:09.9097042Z [2023-03-08 09:16:09,909] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:23.9089011Z Traceback (most recent call last): 2023-03-08T09:16:23.9089732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 595, in 2023-03-08T09:16:23.9090228Z huggingface_main() 2023-03-08T09:16:23.9094154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 591, in huggingface_main 2023-03-08T09:16:23.9094603Z main(HuggingfaceRunner()) 2023-03-08T09:16:23.9095095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1913, in main 2023-03-08T09:16:23.9095563Z return maybe_fresh_cache( 2023-03-08T09:16:23.9096187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 984, in inner 2023-03-08T09:16:23.9096750Z return fn(*args, **kwargs) 2023-03-08T09:16:23.9097258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2290, in run 2023-03-08T09:16:23.9097867Z runner.run_one_model( 2023-03-08T09:16:23.9098235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1467, in run_one_model 2023-03-08T09:16:23.9101349Z status = self.run_performance_test( 2023-03-08T09:16:23.9102207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1440, in run_performance_test 2023-03-08T09:16:23.9103884Z results.append(experiment(model, example_inputs, **experiment_kwargs)) 2023-03-08T09:16:23.9104698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 624, in speedup_experiment 2023-03-08T09:16:23.9105356Z timings[rep, 0], expected_output = timed( 2023-03-08T09:16:23.9105863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 432, in timed 2023-03-08T09:16:23.9106622Z result = model_iter_fn(model, example_inputs, collect_outputs=collect_outputs) 2023-03-08T09:16:23.9107641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in forward_and_backward_pass 2023-03-08T09:16:23.9108674Z pred = mod(**cloned_inputs) 2023-03-08T09:16:23.9109464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:16:23.9111311Z return forward_call(*args, **kwargs) 2023-03-08T09:16:23.9112351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 929, in forward 2023-03-08T09:16:23.9113783Z outputs = self.model.decoder( 2023-03-08T09:16:23.9114804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:16:23.9116674Z return forward_call(*args, **kwargs) 2023-03-08T09:16:23.9117838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 693, in forward 2023-03-08T09:16:23.9118404Z layer_outputs = decoder_layer( 2023-03-08T09:16:23.9119421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:16:23.9122341Z return forward_call(*args, **kwargs) 2023-03-08T09:16:23.9123315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 324, in forward 2023-03-08T09:16:23.9123755Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2023-03-08T09:16:23.9124316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:16:23.9126455Z return forward_call(*args, **kwargs) 2023-03-08T09:16:23.9127145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 230, in forward 2023-03-08T09:16:23.9127665Z attn_weights = nn.functional.softmax(attn_weights, dim=-1) 2023-03-08T09:16:23.9128203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 1843, in softmax 2023-03-08T09:16:23.9134823Z ret = input.softmax(dim) 2023-03-08T09:16:23.9135623Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 39.45 GiB total capacity; 22.63 GiB already allocated; 269.25 MiB free; 38.62 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T09:16:24.9436002Z ERROR 2023-03-08T09:16:31.9132306Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T09:16:34.6646872Z cuda train PLBartForCausalLM [2023-03-08 09:16:34,663] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:38.8584515Z [2023-03-08 09:16:38,857] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:42.8562248Z [2023-03-08 09:16:42,855] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:43.9882414Z [2023-03-08 09:16:43,987] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:45.1113048Z [2023-03-08 09:16:45,110] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:46.2422578Z [2023-03-08 09:16:46,241] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:16:47.3738238Z [2023-03-08 09:16:47,373] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:03.2414631Z 1.259x p=0.00 2023-03-08T09:17:14.3628141Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T09:17:23.3721667Z cuda train PLBartForConditionalGeneration [2023-03-08 09:17:23,371] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:29.7154104Z [2023-03-08 09:17:29,714] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:31.4315773Z [2023-03-08 09:17:31,430] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:35.8777337Z [2023-03-08 09:17:35,876] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:37.6752731Z [2023-03-08 09:17:37,674] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:39.4536872Z [2023-03-08 09:17:39,453] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:41.2310290Z [2023-03-08 09:17:41,230] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:17:43.0139026Z [2023-03-08 09:17:43,013] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:05.9410703Z 1.219x p=0.00 2023-03-08T09:18:20.3644012Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T09:18:20.3644531Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T09:18:23.8515805Z cuda train PegasusForCausalLM [2023-03-08 09:18:23,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:25.2153634Z [2023-03-08 09:18:25,214] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:29.0236863Z [2023-03-08 09:18:29,022] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:30.1151061Z [2023-03-08 09:18:30,114] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:31.2058797Z [2023-03-08 09:18:31,205] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:32.2931291Z [2023-03-08 09:18:32,292] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:33.3800922Z [2023-03-08 09:18:33,379] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:34.4698615Z [2023-03-08 09:18:34,469] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:35.5572363Z [2023-03-08 09:18:35,556] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:36.6478028Z [2023-03-08 09:18:36,647] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:37.7402242Z [2023-03-08 09:18:37,739] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:38.8425581Z [2023-03-08 09:18:38,841] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:39.9440497Z [2023-03-08 09:18:39,943] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:18:59.2533497Z 1.001x SAME 2023-03-08T09:19:23.6472114Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T09:19:23.6473436Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T09:19:36.0453422Z cuda train PegasusForConditionalGeneration [2023-03-08 09:19:36,044] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:19:58.1411708Z [2023-03-08 09:19:58,140] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:20:36.0328295Z 1.027x p=0.00 2023-03-08T09:20:41.3948560Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T09:20:44.4068580Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T09:20:57.2292657Z cuda train RobertaForCausalLM [2023-03-08 09:20:57,228] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:21:14.1485481Z [2023-03-08 09:21:14,147] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T09:21:24.0824965Z 1.268x p=0.00 2023-03-08T09:21:32.0904565Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T09:21:44.8609357Z cuda train RobertaForQuestionAnswering [2023-03-08 09:21:44,859] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:00.2619435Z [2023-03-08 09:22:00,260] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T09:22:08.0506472Z 1.266x p=0.00 2023-03-08T09:22:13.9155216Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T09:22:13.9157252Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T09:22:17.2997541Z cuda train Speech2Text2ForCausalLM [2023-03-08 09:22:17,298] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:18.4356012Z [2023-03-08 09:22:18,434] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:22.0710597Z [2023-03-08 09:22:22,070] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:23.1265740Z [2023-03-08 09:22:23,125] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:24.1781254Z [2023-03-08 09:22:24,177] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:25.2342079Z [2023-03-08 09:22:25,233] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:26.2907940Z [2023-03-08 09:22:26,290] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:22:40.8207135Z 1.109x p=0.00 2023-03-08T09:22:47.4515198Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T09:23:03.5999048Z cuda train T5ForConditionalGeneration [2023-03-08 09:23:03,598] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:23:41.4270916Z 1.261x p=0.00 2023-03-08T09:23:48.3544152Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T09:24:04.4606420Z cuda train T5Small [2023-03-08 09:24:04,459] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:24:42.2550555Z 1.257x p=0.00 2023-03-08T09:24:54.4023958Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T09:24:57.4592827Z cuda train TrOCRForCausalLM [2023-03-08 09:24:57,458] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:01.6028243Z [2023-03-08 09:25:01,602] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:05.0256271Z [2023-03-08 09:25:05,024] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:06.1254263Z [2023-03-08 09:25:06,124] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:07.2189662Z [2023-03-08 09:25:07,218] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:08.3154903Z [2023-03-08 09:25:08,314] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:09.4159301Z [2023-03-08 09:25:09,415] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:10.5248447Z [2023-03-08 09:25:10,524] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:11.6625043Z [2023-03-08 09:25:11,661] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:12.7569843Z [2023-03-08 09:25:12,756] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:13.8711107Z [2023-03-08 09:25:13,870] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:14.9784427Z [2023-03-08 09:25:14,977] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:16.0797478Z [2023-03-08 09:25:16,078] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:29.5072435Z Traceback (most recent call last): 2023-03-08T09:25:29.5073111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 595, in 2023-03-08T09:25:29.5073568Z huggingface_main() 2023-03-08T09:25:29.5074195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 591, in huggingface_main 2023-03-08T09:25:29.5074678Z main(HuggingfaceRunner()) 2023-03-08T09:25:29.5075995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1913, in main 2023-03-08T09:25:29.5076673Z return maybe_fresh_cache( 2023-03-08T09:25:29.5077453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 984, in inner 2023-03-08T09:25:29.5078042Z return fn(*args, **kwargs) 2023-03-08T09:25:29.5078356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2290, in run 2023-03-08T09:25:29.5080221Z runner.run_one_model( 2023-03-08T09:25:29.5080709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1467, in run_one_model 2023-03-08T09:25:29.5084058Z status = self.run_performance_test( 2023-03-08T09:25:29.5084666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1440, in run_performance_test 2023-03-08T09:25:29.5088006Z results.append(experiment(model, example_inputs, **experiment_kwargs)) 2023-03-08T09:25:29.5088727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 624, in speedup_experiment 2023-03-08T09:25:29.5089660Z timings[rep, 0], expected_output = timed( 2023-03-08T09:25:29.5090313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 432, in timed 2023-03-08T09:25:29.5092096Z result = model_iter_fn(model, example_inputs, collect_outputs=collect_outputs) 2023-03-08T09:25:29.5093164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 493, in forward_and_backward_pass 2023-03-08T09:25:29.5093803Z self.grad_scaler.scale(loss).backward() 2023-03-08T09:25:29.5094905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward 2023-03-08T09:25:29.5095403Z torch.autograd.backward( 2023-03-08T09:25:29.5096195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 204, in backward 2023-03-08T09:25:29.5096941Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-03-08T09:25:29.5097717Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.54 GiB (GPU 0; 39.45 GiB total capacity; 28.13 GiB already allocated; 1.21 GiB free; 37.67 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T09:25:30.4993889Z ERROR 2023-03-08T09:25:51.0989535Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T09:25:51.0990030Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T09:25:54.7194674Z cuda train XGLMForCausalLM [2023-03-08 09:25:54,718] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:25:55.9465716Z [2023-03-08 09:25:55,945] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:01.5161571Z [2023-03-08 09:26:01,515] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:02.7186330Z [2023-03-08 09:26:02,717] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:03.9008715Z [2023-03-08 09:26:03,900] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:05.0855406Z [2023-03-08 09:26:05,084] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:06.2634706Z [2023-03-08 09:26:06,262] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:07.4395270Z [2023-03-08 09:26:07,438] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:08.6205958Z [2023-03-08 09:26:08,619] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:09.8030664Z [2023-03-08 09:26:09,802] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:10.9947925Z [2023-03-08 09:26:10,993] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:12.1869721Z [2023-03-08 09:26:12,186] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:13.3745624Z [2023-03-08 09:26:13,373] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:14.5681687Z [2023-03-08 09:26:14,567] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:15.7594771Z [2023-03-08 09:26:15,758] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:16.9672431Z [2023-03-08 09:26:16,966] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:18.1734870Z [2023-03-08 09:26:18,172] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:19.3672929Z [2023-03-08 09:26:19,366] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:20.5668429Z [2023-03-08 09:26:20,566] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:21.7555914Z [2023-03-08 09:26:21,754] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:22.9474713Z [2023-03-08 09:26:22,946] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:24.1514602Z [2023-03-08 09:26:24,150] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:25.3599841Z [2023-03-08 09:26:25,359] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:26.5641874Z [2023-03-08 09:26:26,563] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:27.7647985Z [2023-03-08 09:26:27,764] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:26:58.7063581Z 1.167x p=0.00 2023-03-08T09:27:11.8781168Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T09:27:46.4798621Z cuda train XLNetLMHeadModel [2023-03-08 09:27:46,478] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:27:46.5008175Z [2023-03-08 09:27:46,500] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T09:28:21.9106813Z [2023-03-08 09:28:21,909] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T09:29:13.5673859Z 1.504x p=0.00 2023-03-08T09:29:22.8215503Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T09:29:40.9678847Z cuda train YituTechConvBert [2023-03-08 09:29:40,966] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:30:10.7972161Z [2023-03-08 09:30:10,795] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T09:30:27.9233713Z 1.213x p=0.00 2023-03-08T09:30:29.5606586Z speedup gmean=1.18x mean=1.189x 2023-03-08T09:30:29.5616158Z abs_latency gmean=nanx mean=113.867x 2023-03-08T09:30:29.5617389Z compilation_latency mean=50.349 seconds 2023-03-08T09:30:29.5619485Z compression_ratio mean=0.817x 2023-03-08T09:30:30.0563685Z + python benchmarks/dynamo/huggingface.py --performance --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_float32_training_cuda_performance.csv --training --inductor --disable-cudagraphs --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer --cold-start-latency 2023-03-08T09:30:42.8855557Z WARNING:__main__:Running smaller batch size=4 for AlbertForMaskedLM, orig batch_size=8 2023-03-08T09:31:37.2734313Z cuda train AlbertForMaskedLM 1.255x p=0.00 2023-03-08T09:31:47.3393490Z WARNING:__main__:Running smaller batch size=4 for AlbertForQuestionAnswering, orig batch_size=8 2023-03-08T09:32:38.3205242Z cuda train AlbertForQuestionAnswering 1.260x p=0.00 2023-03-08T09:32:47.6477037Z WARNING:__main__:Running smaller batch size=4 for AllenaiLongformerBase, orig batch_size=8 2023-03-08T09:32:51.6862387Z cuda train AllenaiLongformerBase [2023-03-08 09:32:51,685] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:02.4700567Z ERROR:common:Backend dynamo failed in warmup() 2023-03-08T09:33:02.4700900Z Traceback (most recent call last): 2023-03-08T09:33:02.4701258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1372, in warmup 2023-03-08T09:33:02.4701853Z fn(model, example_inputs) 2023-03-08T09:33:02.4702577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T09:33:02.4703012Z return fn(*args, **kwargs) 2023-03-08T09:33:02.4707464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T09:33:02.4708236Z cloned_inputs = clone_inputs(inputs) 2023-03-08T09:33:02.4708972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T09:33:02.4709659Z self.optimizer_zero_grad(mod) 2023-03-08T09:33:02.4710359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T09:33:02.4711020Z pred = mod(**cloned_inputs) 2023-03-08T09:33:02.4712098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4712813Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4714003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T09:33:02.4714704Z outputs = self.longformer( 2023-03-08T09:33:02.4715622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4716280Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4717309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T09:33:02.4717989Z encoder_outputs = self.encoder( 2023-03-08T09:33:02.4718912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4719564Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4720579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T09:33:02.4721372Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T09:33:02.4722923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T09:33:02.4723679Z layer_outputs = layer_module( 2023-03-08T09:33:02.4724591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4725249Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4726280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T09:33:02.4726988Z self_attn_outputs = self.attention( 2023-03-08T09:33:02.4727907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4728560Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4729592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T09:33:02.4730276Z self_outputs = self.self( 2023-03-08T09:33:02.4731183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T09:33:02.4732042Z return forward_call(*args, **kwargs) 2023-03-08T09:33:02.4733132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T09:33:02.4733905Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T09:33:02.4735036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T09:33:02.4735845Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T09:33:02.4737015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T09:33:02.4737934Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T09:33:02.4738999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T09:33:02.4739685Z return callback(frame, cache_size, hooks) 2023-03-08T09:33:02.4740639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T09:33:02.4741355Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T09:33:02.4742302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T09:33:02.4742968Z return fn(*args, **kwargs) 2023-03-08T09:33:02.4743924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T09:33:02.4744562Z return _compile( 2023-03-08T09:33:02.4745417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T09:33:02.4746016Z r = func(*args, **kwargs) 2023-03-08T09:33:02.4746909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T09:33:02.4747781Z out_code = transform_code_object(code, transform) 2023-03-08T09:33:02.4748854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T09:33:02.4749628Z transformations(instructions, code_options) 2023-03-08T09:33:02.4750613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T09:33:02.4751219Z tracer.run() 2023-03-08T09:33:02.4752063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T09:33:02.4752685Z super().run() 2023-03-08T09:33:02.4753551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T09:33:02.4754381Z and self.step() 2023-03-08T09:33:02.4755232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T09:33:02.4755875Z getattr(self, inst.opname)(inst) 2023-03-08T09:33:02.4756800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T09:33:02.4757511Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T09:33:02.4758486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T09:33:02.4759245Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T09:33:02.4760267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T09:33:02.4760972Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T09:33:02.4761877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T09:33:02.4762727Z r = func(*args, **kwargs) 2023-03-08T09:33:02.4763663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T09:33:02.4764467Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T09:33:02.4765538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T09:33:02.4766231Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T09:33:02.4767170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T09:33:02.4767808Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T09:33:02.4768731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T09:33:02.4769344Z return compile_fx(*args, **kwargs) 2023-03-08T09:33:02.4770233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T09:33:02.4770851Z return aot_autograd( 2023-03-08T09:33:02.4771715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T09:33:02.4772501Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T09:33:02.4773511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T09:33:02.4774211Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T09:33:02.4775097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T09:33:02.4775697Z r = func(*args, **kwargs) 2023-03-08T09:33:02.4776687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T09:33:02.4777451Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T09:33:02.4778488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T09:33:02.4779217Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T09:33:02.4780215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T09:33:02.4780984Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T09:33:02.4781967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T09:33:02.4782604Z r = func(*args, **kwargs) 2023-03-08T09:33:02.4783634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T09:33:02.4784245Z return inner_compile( 2023-03-08T09:33:02.4785139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T09:33:02.4785827Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T09:33:02.4786708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T09:33:02.4787796Z return fn(*args, **kwargs) 2023-03-08T09:33:02.4788283Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T09:33:02.4788734Z return func(*args, **kwds) 2023-03-08T09:33:02.4789672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T09:33:02.4790317Z graph.run(*example_inputs) 2023-03-08T09:33:02.4791195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T09:33:02.4791781Z r = func(*args, **kwargs) 2023-03-08T09:33:02.4792909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T09:33:02.4793519Z return super().run(*args) 2023-03-08T09:33:02.4794349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T09:33:02.4794983Z self.env[node] = self.run_node(node) 2023-03-08T09:33:02.4795860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T09:33:02.4796473Z result = super().run_node(n) 2023-03-08T09:33:02.4797305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T09:33:02.4797895Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T09:33:02.4798756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T09:33:02.4799418Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T09:33:02.4800262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T09:33:02.4800848Z out = lowerings[target](*args, **kwargs) 2023-03-08T09:33:02.4801701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T09:33:02.4802228Z validate_ir(out) 2023-03-08T09:33:02.4803149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T09:33:02.4803740Z _check_tensorbox(node_or_nodes) 2023-03-08T09:33:02.4804601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T09:33:02.4805195Z assert isinstance( 2023-03-08T09:33:02.4805912Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T09:33:02.4807102Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T09:33:02.4807878Z target: aten._local_scalar_dense.default 2023-03-08T09:33:02.4808348Z args[0]: TensorBox(StorageBox( 2023-03-08T09:33:02.4808748Z Pointwise( 2023-03-08T09:33:02.4809144Z 'cpu', 2023-03-08T09:33:02.4809488Z torch.int64, 2023-03-08T09:33:02.4809877Z def inner_fn(index): 2023-03-08T09:33:02.4810327Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T09:33:02.4810838Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T09:33:02.4811329Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T09:33:02.4811741Z return tmp2 2023-03-08T09:33:02.4812094Z , 2023-03-08T09:33:02.4812487Z ranges=(), 2023-03-08T09:33:02.4812830Z origins={div} 2023-03-08T09:33:02.4837517Z ) 2023-03-08T09:33:02.4837977Z )) 2023-03-08T09:33:02.4838189Z 2023-03-08T09:33:02.4838198Z 2023-03-08T09:33:02.4838749Z You can suppress this exception and fall back to eager by setting: 2023-03-08T09:33:02.4839396Z torch._dynamo.config.suppress_errors = True 2023-03-08T09:33:02.4839734Z 2023-03-08T09:33:03.4423734Z ERROR 2023-03-08T09:33:14.1144690Z WARNING:__main__:Running smaller batch size=4 for BartForCausalLM, orig batch_size=8 2023-03-08T09:33:16.8517882Z cuda train BartForCausalLM [2023-03-08 09:33:16,850] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:20.8918364Z [2023-03-08 09:33:20,891] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:23.8795076Z [2023-03-08 09:33:23,878] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:25.0232505Z [2023-03-08 09:33:25,022] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:25.9787710Z [2023-03-08 09:33:25,978] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:26.9333322Z [2023-03-08 09:33:26,932] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:27.8821039Z [2023-03-08 09:33:27,881] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:28.8328707Z [2023-03-08 09:33:28,832] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:29.9546420Z [2023-03-08 09:33:29,954] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:30.9067496Z [2023-03-08 09:33:30,906] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:31.8559038Z [2023-03-08 09:33:31,855] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:32.8078413Z [2023-03-08 09:33:32,807] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:33.7566643Z [2023-03-08 09:33:33,756] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:33:52.0764009Z 1.175x p=0.00 2023-03-08T09:34:11.6005526Z WARNING:__main__:Running smaller batch size=2 for BartForConditionalGeneration, orig batch_size=4 2023-03-08T09:34:38.2484873Z cuda train BartForConditionalGeneration [2023-03-08 09:34:38,247] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:35:23.3841884Z 1.114x p=0.00 2023-03-08T09:35:32.1722484Z WARNING:__main__:Running smaller batch size=16 for BertForMaskedLM, orig batch_size=32 2023-03-08T09:35:44.9893951Z cuda train BertForMaskedLM [2023-03-08 09:35:44,988] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:36:11.9003872Z 1.190x p=0.00 2023-03-08T09:36:19.8146693Z WARNING:__main__:Running smaller batch size=16 for BertForQuestionAnswering, orig batch_size=32 2023-03-08T09:36:32.4422561Z cuda train BertForQuestionAnswering [2023-03-08 09:36:32,441] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:36:57.2185007Z 1.262x p=0.00 2023-03-08T09:37:59.1311568Z WARNING:__main__:Running smaller batch size=4 for BlenderbotForCausalLM, orig batch_size=32 2023-03-08T09:38:01.8619830Z cuda train BlenderbotForCausalLM [2023-03-08 09:38:01,861] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:03.3363506Z [2023-03-08 09:38:03,335] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:08.3701252Z [2023-03-08 09:38:08,369] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:09.3330624Z [2023-03-08 09:38:09,332] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:10.2864497Z [2023-03-08 09:38:10,285] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:11.2419998Z [2023-03-08 09:38:11,241] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:12.3517707Z [2023-03-08 09:38:12,351] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:13.3085746Z [2023-03-08 09:38:13,307] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:14.2611388Z [2023-03-08 09:38:14,260] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:15.2160733Z [2023-03-08 09:38:15,215] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:16.3372899Z [2023-03-08 09:38:16,336] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:17.3078819Z [2023-03-08 09:38:17,307] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:18.2681984Z [2023-03-08 09:38:18,267] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:19.2208857Z [2023-03-08 09:38:19,220] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:20.1749010Z [2023-03-08 09:38:20,174] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:21.3181369Z [2023-03-08 09:38:21,317] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:22.2839557Z [2023-03-08 09:38:22,283] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:23.2447786Z [2023-03-08 09:38:23,244] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:24.1954150Z [2023-03-08 09:38:24,194] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:25.1406275Z [2023-03-08 09:38:25,139] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:26.2776697Z [2023-03-08 09:38:26,277] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:27.2363515Z [2023-03-08 09:38:27,235] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:28.2036891Z [2023-03-08 09:38:28,202] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:29.1556980Z [2023-03-08 09:38:29,155] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:30.1087627Z [2023-03-08 09:38:30,108] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:38:58.8985800Z 0.984x p=0.00 2023-03-08T09:39:06.1072071Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForCausalLM, orig batch_size=256 2023-03-08T09:39:08.5123075Z cuda train BlenderbotSmallForCausalLM [2023-03-08 09:39:08,511] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:10.0688199Z [2023-03-08 09:39:10,068] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:13.4821562Z [2023-03-08 09:39:13,481] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:14.6170052Z [2023-03-08 09:39:14,616] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:15.5671724Z [2023-03-08 09:39:15,566] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:16.5163654Z [2023-03-08 09:39:16,515] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:17.4669136Z [2023-03-08 09:39:17,466] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:18.5864465Z [2023-03-08 09:39:18,585] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:19.5429562Z [2023-03-08 09:39:19,542] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:39:33.6925360Z 1.053x p=0.00 2023-03-08T09:39:43.0284947Z WARNING:__main__:Running smaller batch size=64 for BlenderbotSmallForConditionalGeneration, orig batch_size=128 2023-03-08T09:40:01.3209595Z cuda train BlenderbotSmallForConditionalGeneration [2023-03-08 09:40:01,319] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:40:34.8925563Z 1.091x p=0.00 2023-03-08T09:40:43.5414364Z WARNING:__main__:Running smaller batch size=16 for CamemBert, orig batch_size=32 2023-03-08T09:40:56.5185020Z cuda train CamemBert [2023-03-08 09:40:56,517] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:41:21.3607505Z 1.190x p=0.00 2023-03-08T09:41:30.1715936Z WARNING:__main__:Running smaller batch size=4 for DebertaForMaskedLM, orig batch_size=32 2023-03-08T09:41:33.6002415Z cuda train DebertaForMaskedLM [2023-03-08 09:41:33,599] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:41:40.2029957Z [2023-03-08 09:41:40,202] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:42:11.0726734Z 0.847x p=0.00 2023-03-08T09:42:19.3186341Z WARNING:__main__:Running smaller batch size=8 for DebertaForQuestionAnswering, orig batch_size=32 2023-03-08T09:42:22.5395311Z cuda train DebertaForQuestionAnswering [2023-03-08 09:42:22,538] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:42:27.1970198Z [2023-03-08 09:42:27,196] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:43:00.2001995Z 1.020x p=0.00 2023-03-08T09:43:27.9556822Z WARNING:__main__:Running smaller batch size=1 for DebertaV2ForMaskedLM, orig batch_size=8 2023-03-08T09:43:33.7077452Z cuda train DebertaV2ForMaskedLM [2023-03-08 09:43:33,706] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:43:35.7227530Z [2023-03-08 09:43:35,722] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:44:00.6527553Z [2023-03-08 09:44:00,651] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T09:44:00.6528375Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T09:44:00.6528791Z reasons: ___check_obj_id(self, 140145681248512) 2023-03-08T09:44:00.6529217Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T09:44:24.9756682Z 0.635x p=0.00 2023-03-08T09:44:50.0521683Z WARNING:__main__:Running smaller batch size=2 for DebertaV2ForQuestionAnswering, orig batch_size=8 2023-03-08T09:44:53.5079054Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 09:44:53,506] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:44:55.6168479Z [2023-03-08 09:44:55,616] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:45:20.8556332Z [2023-03-08 09:45:20,854] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) 2023-03-08T09:45:20.8557359Z function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:250) 2023-03-08T09:45:20.8557889Z reasons: ___check_obj_id(self, 140078541549680) 2023-03-08T09:45:20.8558579Z to diagnose recompilation issues, see https://pytorch.org/docs/master/dynamo/troubleshooting.html. 2023-03-08T09:45:46.9847781Z 0.714x p=0.00 2023-03-08T09:45:54.4403769Z WARNING:__main__:Running smaller batch size=128 for DistilBertForMaskedLM, orig batch_size=256 2023-03-08T09:45:54.4404788Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T09:46:01.5052762Z cuda train DistilBertForMaskedLM [2023-03-08 09:46:01,504] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:46:29.1345957Z 1.056x p=0.00 2023-03-08T09:46:35.9101076Z WARNING:__main__:Running smaller batch size=256 for DistilBertForQuestionAnswering, orig batch_size=512 2023-03-08T09:46:35.9101626Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T09:46:43.2074860Z cuda train DistilBertForQuestionAnswering [2023-03-08 09:46:43,206] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:47:13.2583086Z 1.142x p=0.00 2023-03-08T09:47:21.2045064Z WARNING:__main__:Running smaller batch size=16 for DistillGPT2, orig batch_size=32 2023-03-08T09:47:28.3083540Z cuda train DistillGPT2 [2023-03-08 09:47:28,307] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:47:48.1948674Z 1.427x p=0.00 2023-03-08T09:47:53.2989576Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T09:47:53.8884034Z WARNING:__main__:Running smaller batch size=32 for ElectraForCausalLM, orig batch_size=64 2023-03-08T09:48:06.2221657Z cuda train ElectraForCausalLM [2023-03-08 09:48:06,221] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:48:45.6938819Z 1.439x p=0.00 2023-03-08T09:48:51.4489835Z WARNING:__main__:Running smaller batch size=64 for ElectraForQuestionAnswering, orig batch_size=128 2023-03-08T09:49:04.2552539Z cuda train ElectraForQuestionAnswering [2023-03-08 09:49:04,253] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:49:34.3180056Z 1.422x p=0.00 2023-03-08T09:49:43.1491314Z WARNING:__main__:Running smaller batch size=4 for GPT2ForSequenceClassification, orig batch_size=8 2023-03-08T09:49:55.0633300Z cuda train GPT2ForSequenceClassification [2023-03-08 09:49:55,062] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:17.4897809Z 1.668x p=0.00 2023-03-08T09:50:25.2939418Z WARNING:__main__:Running smaller batch size=16 for GoogleFnet, orig batch_size=32 2023-03-08T09:50:27.6155708Z cuda train GoogleFnet [2023-03-08 09:50:27,614] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:31.0293147Z [2023-03-08 09:50:31,028] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:32.0777482Z [2023-03-08 09:50:32,077] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:32.5881116Z [2023-03-08 09:50:32,587] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:33.0919471Z [2023-03-08 09:50:33,091] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:33.7707035Z [2023-03-08 09:50:33,770] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:34.2885029Z [2023-03-08 09:50:34,287] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:34.7926027Z [2023-03-08 09:50:34,791] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:35.3097934Z [2023-03-08 09:50:35,309] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:35.8193195Z [2023-03-08 09:50:35,818] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:36.3293375Z [2023-03-08 09:50:36,328] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:36.8411656Z [2023-03-08 09:50:36,840] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:37.3506992Z [2023-03-08 09:50:37,349] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:50:52.1760840Z 1.545x p=0.00 2023-03-08T09:51:00.2452876Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForMaskedLM, orig batch_size=32 2023-03-08T09:51:13.6101629Z cuda train LayoutLMForMaskedLM [2023-03-08 09:51:13,608] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:51:40.8144653Z 1.197x p=0.00 2023-03-08T09:51:48.7595614Z WARNING:__main__:Running smaller batch size=16 for LayoutLMForSequenceClassification, orig batch_size=32 2023-03-08T09:52:01.8027516Z cuda train LayoutLMForSequenceClassification [2023-03-08 09:52:01,801] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:52:32.7348348Z 1.280x p=0.00 2023-03-08T09:52:58.4152155Z WARNING:__main__:Running smaller batch size=16 for M2M100ForConditionalGeneration, orig batch_size=64 2023-03-08T09:52:58.4152720Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T09:53:10.3433488Z cuda train M2M100ForConditionalGeneration [2023-03-08 09:53:10,342] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:25.9794087Z [2023-03-08 09:53:25,978] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:27.4397314Z [2023-03-08 09:53:27,439] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:33.2159912Z [2023-03-08 09:53:33,215] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:35.0471016Z [2023-03-08 09:53:35,046] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:36.6436081Z [2023-03-08 09:53:36,642] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:38.4424171Z [2023-03-08 09:53:38,441] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:40.0512092Z [2023-03-08 09:53:40,050] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:41.6540822Z [2023-03-08 09:53:41,653] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:43.4721551Z [2023-03-08 09:53:43,471] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:45.0745105Z [2023-03-08 09:53:45,073] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:46.6729901Z [2023-03-08 09:53:46,672] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:48.4908971Z [2023-03-08 09:53:48,490] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:53:50.1039139Z [2023-03-08 09:53:50,103] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:33.8070622Z 1.106x p=0.00 2023-03-08T09:54:46.2205057Z WARNING:__main__:Running smaller batch size=4 for MBartForCausalLM, orig batch_size=8 2023-03-08T09:54:49.0320239Z cuda train MBartForCausalLM [2023-03-08 09:54:49,031] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:53.0878289Z [2023-03-08 09:54:53,086] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:56.0695040Z [2023-03-08 09:54:56,068] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:57.1974275Z [2023-03-08 09:54:57,196] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:58.1523583Z [2023-03-08 09:54:58,151] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:54:59.1072685Z [2023-03-08 09:54:59,106] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:00.0592923Z [2023-03-08 09:55:00,058] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:01.1795571Z [2023-03-08 09:55:01,178] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:02.1374283Z [2023-03-08 09:55:02,136] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:03.0975223Z [2023-03-08 09:55:03,096] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:04.0510551Z [2023-03-08 09:55:04,050] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:05.0057042Z [2023-03-08 09:55:05,004] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:06.1359997Z [2023-03-08 09:55:06,135] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:55:24.7500222Z 1.168x p=0.00 2023-03-08T09:55:44.2295363Z WARNING:__main__:Running smaller batch size=2 for MBartForConditionalGeneration, orig batch_size=4 2023-03-08T09:56:11.1090204Z cuda train MBartForConditionalGeneration [2023-03-08 09:56:11,107] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:56:59.4301023Z 1.111x p=0.00 2023-03-08T09:57:10.6328667Z WARNING:__main__:Running smaller batch size=16 for MT5ForConditionalGeneration, orig batch_size=32 2023-03-08T09:57:10.6329388Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T09:57:33.1908162Z cuda train MT5ForConditionalGeneration [2023-03-08 09:57:33,189] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T09:59:19.5904352Z 1.407x p=0.00 2023-03-08T09:59:25.2097019Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T09:59:33.2969755Z WARNING:__main__:Running smaller batch size=4 for MegatronBertForCausalLM, orig batch_size=16 2023-03-08T09:59:56.4668349Z cuda train MegatronBertForCausalLM [2023-03-08 09:59:56,465] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:00:35.1879008Z 1.163x p=0.00 2023-03-08T10:00:48.5058042Z WARNING:__main__:Running smaller batch size=8 for MegatronBertForQuestionAnswering, orig batch_size=16 2023-03-08T10:01:11.9117878Z cuda train MegatronBertForQuestionAnswering [2023-03-08 10:01:11,910] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:01:54.4066016Z 1.210x p=0.00 2023-03-08T10:02:01.1429028Z WARNING:__main__:Running smaller batch size=64 for MobileBertForMaskedLM, orig batch_size=256 2023-03-08T10:02:42.1871844Z cuda train MobileBertForMaskedLM [2023-03-08 10:02:42,185] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:04:01.7885171Z 1.214x p=0.00 2023-03-08T10:04:09.3318200Z WARNING:__main__:Running smaller batch size=128 for MobileBertForQuestionAnswering, orig batch_size=256 2023-03-08T10:04:50.6092016Z cuda train MobileBertForQuestionAnswering [2023-03-08 10:04:50,607] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:06.0089457Z 1.326x p=0.00 2023-03-08T10:06:17.2953195Z WARNING:__main__:Running smaller batch size=2 for OPTForCausalLM, orig batch_size=4 2023-03-08T10:06:21.7076457Z cuda train OPTForCausalLM [2023-03-08 10:06:21,706] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:25.1728690Z [2023-03-08 10:06:25,172] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:26.3080963Z [2023-03-08 10:06:26,307] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:27.2569705Z [2023-03-08 10:06:27,256] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:28.2042915Z [2023-03-08 10:06:28,203] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:29.1557860Z [2023-03-08 10:06:29,155] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:30.2591389Z [2023-03-08 10:06:30,258] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:31.2145360Z [2023-03-08 10:06:31,213] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:32.1665890Z [2023-03-08 10:06:32,165] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:33.1181231Z [2023-03-08 10:06:33,117] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:34.2474099Z [2023-03-08 10:06:34,246] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:35.2107288Z [2023-03-08 10:06:35,209] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:06:55.1700288Z 1.768x p=0.00 2023-03-08T10:07:03.0977191Z WARNING:__main__:Running smaller batch size=8 for PLBartForCausalLM, orig batch_size=16 2023-03-08T10:07:05.8397895Z cuda train PLBartForCausalLM [2023-03-08 10:07:05,838] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:09.8318718Z [2023-03-08 10:07:09,830] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:13.6501892Z [2023-03-08 10:07:13,649] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:14.8386177Z [2023-03-08 10:07:14,837] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:15.8354741Z [2023-03-08 10:07:15,834] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:16.8317323Z [2023-03-08 10:07:16,831] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:17.8211144Z [2023-03-08 10:07:17,820] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:32.3490057Z 1.295x p=0.00 2023-03-08T10:07:43.4042032Z WARNING:__main__:Running smaller batch size=4 for PLBartForConditionalGeneration, orig batch_size=8 2023-03-08T10:07:52.3986686Z cuda train PLBartForConditionalGeneration [2023-03-08 10:07:52,397] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:58.3564114Z [2023-03-08 10:07:58,355] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:07:59.9251811Z [2023-03-08 10:07:59,924] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:04.3893164Z [2023-03-08 10:08:04,388] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:06.0161126Z [2023-03-08 10:08:06,015] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:07.8444913Z [2023-03-08 10:08:07,843] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:09.4698782Z [2023-03-08 10:08:09,469] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:11.2695332Z [2023-03-08 10:08:11,268] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:32.5996375Z 1.254x p=0.00 2023-03-08T10:08:46.8768111Z WARNING:__main__:Running smaller batch size=32 for PegasusForCausalLM, orig batch_size=128 2023-03-08T10:08:46.8768623Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:08:50.1011508Z cuda train PegasusForCausalLM [2023-03-08 10:08:50,100] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:51.3484720Z [2023-03-08 10:08:51,347] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:55.0398575Z [2023-03-08 10:08:55,038] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:56.1777602Z [2023-03-08 10:08:56,177] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:57.1359981Z [2023-03-08 10:08:57,135] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:58.0860282Z [2023-03-08 10:08:58,085] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:08:59.0427521Z [2023-03-08 10:08:59,042] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:00.1734834Z [2023-03-08 10:09:00,172] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:01.1328469Z [2023-03-08 10:09:01,132] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:02.1011272Z [2023-03-08 10:09:02,100] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:03.0753575Z [2023-03-08 10:09:03,074] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:04.2202583Z [2023-03-08 10:09:04,219] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:05.1822401Z [2023-03-08 10:09:05,181] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:09:22.2687363Z 1.044x p=0.00 2023-03-08T10:09:46.5641674Z WARNING:__main__:Running smaller batch size=32 for PegasusForConditionalGeneration, orig batch_size=64 2023-03-08T10:09:46.5642617Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:09:58.7951439Z cuda train PegasusForConditionalGeneration [2023-03-08 10:09:58,794] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:10:20.3572805Z [2023-03-08 10:10:20,355] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:10:56.0247533Z 1.046x p=0.00 2023-03-08T10:11:01.4528550Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T10:11:04.4785004Z WARNING:__main__:Running smaller batch size=16 for RobertaForCausalLM, orig batch_size=32 2023-03-08T10:11:17.3132683Z cuda train RobertaForCausalLM [2023-03-08 10:11:17,312] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:11:44.0312605Z 1.265x p=0.00 2023-03-08T10:11:51.9251421Z WARNING:__main__:Running smaller batch size=16 for RobertaForQuestionAnswering, orig batch_size=32 2023-03-08T10:12:04.6411947Z cuda train RobertaForQuestionAnswering [2023-03-08 10:12:04,640] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:27.7970435Z 1.261x p=0.00 2023-03-08T10:12:33.5340082Z WARNING:__main__:Running smaller batch size=256 for Speech2Text2ForCausalLM, orig batch_size=1024 2023-03-08T10:12:33.5340611Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:12:36.6328633Z cuda train Speech2Text2ForCausalLM [2023-03-08 10:12:36,632] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:37.6536993Z [2023-03-08 10:12:37,652] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:41.2268178Z [2023-03-08 10:12:41,225] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:42.2842963Z [2023-03-08 10:12:42,283] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:43.2078014Z [2023-03-08 10:12:43,207] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:44.1313303Z [2023-03-08 10:12:44,130] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:45.0550481Z [2023-03-08 10:12:45,054] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:12:58.3187380Z 1.176x p=0.00 2023-03-08T10:13:05.0141768Z WARNING:__main__:Running smaller batch size=4 for T5ForConditionalGeneration, orig batch_size=8 2023-03-08T10:13:21.0460240Z cuda train T5ForConditionalGeneration [2023-03-08 10:13:21,044] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:13:58.3094253Z 1.324x p=0.00 2023-03-08T10:14:05.4163862Z WARNING:__main__:Running smaller batch size=4 for T5Small, orig batch_size=8 2023-03-08T10:14:21.5451647Z cuda train T5Small [2023-03-08 10:14:21,544] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:14:58.3456757Z 1.324x p=0.00 2023-03-08T10:15:10.4196615Z WARNING:__main__:Running smaller batch size=32 for TrOCRForCausalLM, orig batch_size=64 2023-03-08T10:15:13.4351500Z cuda train TrOCRForCausalLM [2023-03-08 10:15:13,434] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:17.4059756Z [2023-03-08 10:15:17,405] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:20.6983794Z [2023-03-08 10:15:20,697] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:21.8343593Z [2023-03-08 10:15:21,833] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:22.7837987Z [2023-03-08 10:15:22,783] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:23.7309551Z [2023-03-08 10:15:23,730] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:24.6833306Z [2023-03-08 10:15:24,682] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:25.6278558Z [2023-03-08 10:15:25,627] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:26.7393811Z [2023-03-08 10:15:26,738] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:27.6857565Z [2023-03-08 10:15:27,685] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:28.6322360Z [2023-03-08 10:15:28,631] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:29.5798217Z [2023-03-08 10:15:29,579] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:30.5266173Z [2023-03-08 10:15:30,525] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:15:51.5525710Z 1.067x p=0.00 2023-03-08T10:16:13.1703336Z WARNING:__main__:Running smaller batch size=8 for XGLMForCausalLM, orig batch_size=32 2023-03-08T10:16:13.1703896Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:16:16.4529036Z cuda train XGLMForCausalLM [2023-03-08 10:16:16,451] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:17.5643873Z [2023-03-08 10:16:17,563] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:23.0955039Z [2023-03-08 10:16:23,094] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:24.3243190Z [2023-03-08 10:16:24,323] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:25.3718019Z [2023-03-08 10:16:25,371] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:26.4223258Z [2023-03-08 10:16:26,421] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:27.6350540Z [2023-03-08 10:16:27,634] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:28.6933385Z [2023-03-08 10:16:28,692] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:29.7430696Z [2023-03-08 10:16:29,742] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:30.7830767Z [2023-03-08 10:16:30,781] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:31.9969002Z [2023-03-08 10:16:31,996] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:33.0450572Z [2023-03-08 10:16:33,044] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:34.0927677Z [2023-03-08 10:16:34,092] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:35.1412813Z [2023-03-08 10:16:35,140] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:36.3653986Z [2023-03-08 10:16:36,364] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:37.4146920Z [2023-03-08 10:16:37,414] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:38.4667395Z [2023-03-08 10:16:38,465] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:39.5157251Z [2023-03-08 10:16:39,514] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:40.7555943Z [2023-03-08 10:16:40,754] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:41.8059852Z [2023-03-08 10:16:41,805] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:42.8526949Z [2023-03-08 10:16:42,852] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:43.8958846Z [2023-03-08 10:16:43,895] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:45.1268890Z [2023-03-08 10:16:45,126] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:46.1669091Z [2023-03-08 10:16:46,166] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:16:47.2124954Z [2023-03-08 10:16:47,211] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:17:14.0330498Z 0.991x p=0.10 2023-03-08T10:17:27.1489636Z WARNING:__main__:Running smaller batch size=8 for XLNetLMHeadModel, orig batch_size=16 2023-03-08T10:18:02.0159662Z cuda train XLNetLMHeadModel [2023-03-08 10:18:02,014] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:18:02.0369612Z [2023-03-08 10:18:02,036] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T10:19:28.9991525Z 1.502x p=0.00 2023-03-08T10:19:38.3509343Z WARNING:__main__:Running smaller batch size=16 for YituTechConvBert, orig batch_size=32 2023-03-08T10:19:56.5717209Z cuda train YituTechConvBert [2023-03-08 10:19:56,570] torch._inductor.utils: [WARNING] using triton random, expect difference from eager 2023-03-08T10:20:42.2431690Z 1.222x p=0.00 2023-03-08T10:20:43.6463701Z speedup gmean=1.21x mean=1.218x 2023-03-08T10:20:43.6475192Z abs_latency gmean=nanx mean=114.241x 2023-03-08T10:20:43.6475719Z compilation_latency mean=44.372 seconds 2023-03-08T10:20:43.6478161Z compression_ratio mean=0.983x 2023-03-08T10:20:44.1425954Z + python benchmarks/dynamo/huggingface.py --accuracy --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/eager_huggingface_float32_training_cuda_accuracy.csv --training --backend=eager --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer 2023-03-08T10:21:03.8812778Z cuda train AlbertForMaskedLM PASS 2023-03-08T10:21:20.0457351Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T10:21:41.4216985Z cuda train AllenaiLongformerBase PASS 2023-03-08T10:21:58.2616998Z cuda train BartForCausalLM PASS 2023-03-08T10:22:28.4283399Z cuda train BartForConditionalGeneration PASS 2023-03-08T10:22:40.8862264Z cuda train BertForMaskedLM PASS 2023-03-08T10:22:52.8544367Z cuda train BertForQuestionAnswering PASS 2023-03-08T10:23:54.3620681Z cuda train BlenderbotForCausalLM PASS 2023-03-08T10:24:04.5411471Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T10:24:20.1184794Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T10:24:32.7850152Z cuda train CamemBert PASS 2023-03-08T10:24:47.4070489Z cuda train DebertaForMaskedLM PASS 2023-03-08T10:25:01.4062996Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T10:25:29.0663824Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T10:26:06.3789622Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T10:26:13.3189522Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T10:26:15.7295382Z cuda train DistilBertForMaskedLM PASS 2023-03-08T10:26:22.0580787Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T10:26:24.4246858Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T10:26:34.3902920Z cuda train DistillGPT2 PASS 2023-03-08T10:26:39.0479706Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T10:26:44.3758849Z cuda train ElectraForCausalLM PASS 2023-03-08T10:26:54.3320179Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T10:27:07.0780740Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T10:27:17.1839866Z cuda train GoogleFnet PASS 2023-03-08T10:27:30.1218849Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T10:27:42.7077499Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T10:28:08.0158087Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:28:18.9110078Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T10:28:35.8317629Z cuda train MBartForCausalLM PASS 2023-03-08T10:29:06.3152109Z cuda train MBartForConditionalGeneration PASS 2023-03-08T10:29:16.3702215Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:29:24.5281897Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T10:29:29.2753637Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T10:29:46.8055337Z cuda train MegatronBertForCausalLM PASS 2023-03-08T10:30:08.6824829Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T10:30:29.9518115Z cuda train MobileBertForMaskedLM PASS 2023-03-08T10:30:50.9202555Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T10:31:05.8086993Z cuda train OPTForCausalLM PASS 2023-03-08T10:31:16.5174711Z cuda train PLBartForCausalLM PASS 2023-03-08T10:31:34.4405039Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T10:31:48.1264623Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:31:52.9181582Z cuda train PegasusForCausalLM PASS 2023-03-08T10:32:16.8033696Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:32:23.3764884Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T10:32:28.1141967Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T10:32:35.8923749Z cuda train RobertaForCausalLM PASS 2023-03-08T10:32:47.9431584Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T10:32:53.1821749Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:32:55.9709035Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T10:33:07.9653639Z cuda train T5ForConditionalGeneration PASS 2023-03-08T10:33:20.0830258Z cuda train T5Small PASS 2023-03-08T10:33:36.3708810Z cuda train TrOCRForCausalLM PASS 2023-03-08T10:33:57.6526118Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:34:06.0479125Z cuda train XGLMForCausalLM PASS 2023-03-08T10:34:29.6061826Z cuda train XLNetLMHeadModel PASS 2023-03-08T10:34:44.1649569Z cuda train YituTechConvBert PASS 2023-03-08T10:34:44.9583121Z accuracy pass_rate=95.65% 2023-03-08T10:34:44.9592783Z calls_captured gmean=432.72x mean=638.217x 2023-03-08T10:34:44.9601772Z unique_graphs gmean=3.35x mean=12.935x 2023-03-08T10:34:44.9609786Z graph_breaks gmean=6.98x mean=13.630x 2023-03-08T10:34:44.9617919Z unique_graph_breaks gmean=4.20x mean=4.935x 2023-03-08T10:34:45.4645109Z + python benchmarks/dynamo/huggingface.py --accuracy --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/aot_eager_huggingface_float32_training_cuda_accuracy.csv --training --backend=aot_eager --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer 2023-03-08T10:35:10.5878850Z cuda train AlbertForMaskedLM PASS 2023-03-08T10:35:32.2710294Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T10:36:19.2755510Z cuda train AllenaiLongformerBase PASS 2023-03-08T10:36:41.5169348Z cuda train BartForCausalLM PASS 2023-03-08T10:37:24.6987893Z cuda train BartForConditionalGeneration PASS 2023-03-08T10:37:42.6031424Z cuda train BertForMaskedLM PASS 2023-03-08T10:37:59.8688177Z cuda train BertForQuestionAnswering PASS 2023-03-08T10:39:01.5129978Z cuda train BlenderbotForCausalLM PASS 2023-03-08T10:39:15.2396854Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T10:39:39.4119417Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T10:39:57.0750166Z cuda train CamemBert PASS 2023-03-08T10:40:18.0865195Z cuda train DebertaForMaskedLM PASS 2023-03-08T10:40:38.4989512Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T10:41:06.2775699Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T10:41:53.9219248Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T10:42:01.0268937Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T10:42:06.0284759Z cuda train DistilBertForMaskedLM PASS 2023-03-08T10:42:12.4625819Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T10:42:17.4842561Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T10:42:29.9289317Z cuda train DistillGPT2 PASS 2023-03-08T10:42:34.6403654Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T10:42:45.1394193Z cuda train ElectraForCausalLM PASS 2023-03-08T10:43:00.3162428Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T10:43:17.8589618Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T10:43:30.5042270Z cuda train GoogleFnet PASS 2023-03-08T10:43:48.6510938Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T10:44:06.5110435Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T10:44:31.7393195Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:44:55.4679416Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T10:45:17.5909188Z cuda train MBartForCausalLM PASS 2023-03-08T10:46:00.8516633Z cuda train MBartForConditionalGeneration PASS 2023-03-08T10:46:11.0619509Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:46:29.3961398Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T10:46:34.2322871Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T10:47:01.7734830Z cuda train MegatronBertForCausalLM PASS 2023-03-08T10:47:33.7009855Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T10:48:15.9028300Z cuda train MobileBertForMaskedLM PASS 2023-03-08T10:48:57.8314770Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T10:49:18.1320331Z cuda train OPTForCausalLM PASS 2023-03-08T10:49:31.5358496Z cuda train PLBartForCausalLM PASS 2023-03-08T10:49:56.1638556Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T10:50:10.1367898Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:50:19.9668496Z cuda train PegasusForCausalLM PASS 2023-03-08T10:50:44.0364250Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T10:51:03.4634602Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T10:51:08.3364678Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T10:51:21.2149833Z cuda train RobertaForCausalLM PASS 2023-03-08T10:51:38.5505913Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T10:51:43.7794552Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:51:49.1494133Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T10:52:07.9404724Z cuda train T5ForConditionalGeneration PASS 2023-03-08T10:52:26.8930416Z cuda train T5Small PASS 2023-03-08T10:52:48.4162526Z cuda train TrOCRForCausalLM PASS 2023-03-08T10:53:09.8068985Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T10:53:28.6391518Z cuda train XGLMForCausalLM PASS 2023-03-08T10:54:10.4397956Z cuda train XLNetLMHeadModel PASS 2023-03-08T10:54:33.8186834Z cuda train YituTechConvBert PASS 2023-03-08T10:54:34.6823649Z accuracy pass_rate=95.65% 2023-03-08T10:54:34.6832777Z calls_captured gmean=432.72x mean=638.217x 2023-03-08T10:54:34.6841616Z unique_graphs gmean=3.35x mean=12.935x 2023-03-08T10:54:34.6849019Z graph_breaks gmean=6.98x mean=13.630x 2023-03-08T10:54:34.6856741Z unique_graph_breaks gmean=4.20x mean=4.935x 2023-03-08T10:54:35.1777993Z + python benchmarks/dynamo/huggingface.py --accuracy --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_huggingface_float32_training_cuda_accuracy.csv --training --inductor --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer --cold-start-latency 2023-03-08T10:55:21.0253914Z cuda train AlbertForMaskedLM PASS 2023-03-08T10:56:00.1001693Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T10:56:18.8608284Z cuda train AllenaiLongformerBase [2023-03-08 10:56:18,859] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T10:56:21.0333255Z [2023-03-08 10:56:21,032] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T10:56:25.9468066Z [2023-03-08 10:56:25,945] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T10:56:25.9843110Z ERROR:common:backend='inductor' raised: 2023-03-08T10:56:25.9845767Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T10:56:25.9846257Z target: aten._local_scalar_dense.default 2023-03-08T10:56:25.9846526Z args[0]: TensorBox(StorageBox( 2023-03-08T10:56:25.9846773Z Pointwise( 2023-03-08T10:56:25.9847026Z 'cpu', 2023-03-08T10:56:25.9849014Z torch.int64, 2023-03-08T10:56:25.9849451Z def inner_fn(index): 2023-03-08T10:56:25.9852791Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T10:56:25.9853231Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T10:56:25.9853594Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T10:56:25.9854107Z return tmp2 2023-03-08T10:56:25.9855119Z , 2023-03-08T10:56:25.9855727Z ranges=(), 2023-03-08T10:56:25.9856085Z origins={div} 2023-03-08T10:56:25.9856474Z ) 2023-03-08T10:56:25.9856768Z )) 2023-03-08T10:56:25.9856950Z 2023-03-08T10:56:25.9856962Z 2023-03-08T10:56:25.9857224Z You can suppress this exception and fall back to eager by setting: 2023-03-08T10:56:25.9857904Z torch._dynamo.config.suppress_errors = True 2023-03-08T10:56:25.9858459Z Traceback (most recent call last): 2023-03-08T10:56:25.9858931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T10:56:25.9859317Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T10:56:25.9859957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T10:56:25.9860310Z return fn(*args, **kwargs) 2023-03-08T10:56:25.9860656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T10:56:25.9861047Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T10:56:25.9861462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T10:56:25.9861837Z cloned_inputs = clone_inputs(inputs) 2023-03-08T10:56:25.9862223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T10:56:25.9862598Z self.optimizer_zero_grad(mod) 2023-03-08T10:56:25.9862989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T10:56:25.9863341Z pred = mod(**cloned_inputs) 2023-03-08T10:56:25.9863849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9864257Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9864837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T10:56:25.9865219Z outputs = self.longformer( 2023-03-08T10:56:25.9865721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9866083Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9866636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T10:56:25.9867031Z encoder_outputs = self.encoder( 2023-03-08T10:56:25.9867837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9868204Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9868989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T10:56:25.9869441Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T10:56:25.9870077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T10:56:25.9870470Z layer_outputs = layer_module( 2023-03-08T10:56:25.9870977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9871338Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9871900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T10:56:25.9872287Z self_attn_outputs = self.attention( 2023-03-08T10:56:25.9872795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9873158Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9873798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T10:56:25.9874222Z self_outputs = self.self( 2023-03-08T10:56:25.9874714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T10:56:25.9875077Z return forward_call(*args, **kwargs) 2023-03-08T10:56:25.9875620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T10:56:25.9876041Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T10:56:25.9876648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T10:56:25.9877153Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T10:56:25.9877860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T10:56:25.9878367Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T10:56:25.9878952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T10:56:25.9879330Z return callback(frame, cache_size, hooks) 2023-03-08T10:56:25.9879846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T10:56:25.9880241Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T10:56:25.9880752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T10:56:25.9881094Z return fn(*args, **kwargs) 2023-03-08T10:56:25.9881604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T10:56:25.9881959Z return _compile( 2023-03-08T10:56:25.9882433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T10:56:25.9882758Z r = func(*args, **kwargs) 2023-03-08T10:56:25.9883246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T10:56:25.9883635Z out_code = transform_code_object(code, transform) 2023-03-08T10:56:25.9884216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T10:56:25.9884622Z transformations(instructions, code_options) 2023-03-08T10:56:25.9885149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T10:56:25.9885488Z tracer.run() 2023-03-08T10:56:25.9886046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T10:56:25.9886388Z super().run() 2023-03-08T10:56:25.9886865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T10:56:25.9887199Z and self.step() 2023-03-08T10:56:25.9887713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T10:56:25.9888085Z getattr(self, inst.opname)(inst) 2023-03-08T10:56:25.9888597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T10:56:25.9888985Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T10:56:25.9889537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T10:56:25.9889968Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T10:56:25.9890630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T10:56:25.9891015Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T10:56:25.9891524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T10:56:25.9891864Z r = func(*args, **kwargs) 2023-03-08T10:56:25.9892362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T10:56:25.9892800Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T10:56:25.9893376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T10:56:25.9893793Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T10:56:25.9894326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T10:56:25.9894716Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T10:56:25.9895243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T10:56:25.9895596Z return compile_fx(*args, **kwargs) 2023-03-08T10:56:25.9896102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T10:56:25.9896447Z return aot_autograd( 2023-03-08T10:56:25.9896944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T10:56:25.9897341Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T10:56:25.9897954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T10:56:25.9898362Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T10:56:25.9898866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T10:56:25.9899209Z r = func(*args, **kwargs) 2023-03-08T10:56:25.9899751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T10:56:25.9900187Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T10:56:25.9900741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T10:56:25.9901155Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T10:56:25.9901715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T10:56:25.9902143Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T10:56:25.9902746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T10:56:25.9903092Z r = func(*args, **kwargs) 2023-03-08T10:56:25.9903591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T10:56:25.9903925Z return inner_compile( 2023-03-08T10:56:25.9904417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T10:56:25.9904800Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T10:56:25.9905292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T10:56:25.9905631Z return fn(*args, **kwargs) 2023-03-08T10:56:25.9905951Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T10:56:25.9906259Z return func(*args, **kwds) 2023-03-08T10:56:25.9906753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T10:56:25.9907384Z graph.run(*example_inputs) 2023-03-08T10:56:25.9907921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T10:56:25.9908247Z r = func(*args, **kwargs) 2023-03-08T10:56:25.9908717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T10:56:25.9909058Z return super().run(*args) 2023-03-08T10:56:25.9909534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T10:56:25.9909879Z self.env[node] = self.run_node(node) 2023-03-08T10:56:25.9910373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T10:56:25.9910719Z result = super().run_node(n) 2023-03-08T10:56:25.9911188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T10:56:25.9911569Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T10:56:25.9912092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T10:56:25.9912500Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T10:56:25.9913052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T10:56:25.9913424Z out = lowerings[target](*args, **kwargs) 2023-03-08T10:56:25.9913929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T10:56:25.9914250Z validate_ir(out) 2023-03-08T10:56:25.9914715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T10:56:25.9915066Z _check_tensorbox(node_or_nodes) 2023-03-08T10:56:25.9915548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T10:56:25.9915889Z assert isinstance( 2023-03-08T10:56:25.9916309Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T10:56:25.9916984Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T10:56:25.9917414Z target: aten._local_scalar_dense.default 2023-03-08T10:56:25.9917742Z args[0]: TensorBox(StorageBox( 2023-03-08T10:56:25.9917984Z Pointwise( 2023-03-08T10:56:25.9918216Z 'cpu', 2023-03-08T10:56:25.9918428Z torch.int64, 2023-03-08T10:56:25.9918653Z def inner_fn(index): 2023-03-08T10:56:25.9918905Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T10:56:25.9919198Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T10:56:25.9919480Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T10:56:25.9919800Z return tmp2 2023-03-08T10:56:25.9920022Z , 2023-03-08T10:56:25.9920223Z ranges=(), 2023-03-08T10:56:25.9920422Z origins={div} 2023-03-08T10:56:25.9920625Z ) 2023-03-08T10:56:25.9920810Z )) 2023-03-08T10:56:25.9920928Z 2023-03-08T10:56:25.9920934Z 2023-03-08T10:56:25.9921106Z You can suppress this exception and fall back to eager by setting: 2023-03-08T10:56:25.9921436Z torch._dynamo.config.suppress_errors = True 2023-03-08T10:56:25.9921621Z 2023-03-08T10:56:25.9921797Z TorchDynamo optimized model failed to run because of following error 2023-03-08T10:56:25.9955874Z FAIL 2023-03-08T10:57:08.9688272Z cuda train BartForCausalLM PASS 2023-03-08T10:58:20.7118407Z cuda train BartForConditionalGeneration PASS 2023-03-08T10:58:51.1522481Z cuda train BertForMaskedLM PASS 2023-03-08T10:59:19.5689931Z cuda train BertForQuestionAnswering PASS 2023-03-08T11:00:21.7575538Z cuda train BlenderbotForCausalLM PASS 2023-03-08T11:01:03.8556427Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T11:02:09.0613342Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T11:02:39.1138477Z cuda train CamemBert PASS 2023-03-08T11:02:55.5336109Z cuda train DebertaForMaskedLM [2023-03-08 11:02:55,532] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T11:03:32.4001859Z PASS 2023-03-08T11:03:48.1927728Z cuda train DebertaForQuestionAnswering [2023-03-08 11:03:48,191] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T11:04:23.0479314Z PASS 2023-03-08T11:04:51.1155014Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T11:05:23.9161658Z cuda train DebertaV2ForQuestionAnswering [2023-03-08 11:05:23,914] torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation 2023-03-08T11:06:11.3034149Z ERROR:common:CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 39.45 GiB total capacity; 36.02 GiB already allocated; 9.25 MiB free; 38.83 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T11:06:11.3034853Z Traceback (most recent call last): 2023-03-08T11:06:11.3035218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T11:06:11.3035615Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T11:06:11.3036554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T11:06:11.3036890Z return fn(*args, **kwargs) 2023-03-08T11:06:11.3037240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T11:06:11.3037681Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T11:06:11.3038083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T11:06:11.3038485Z cloned_inputs = clone_inputs(inputs) 2023-03-08T11:06:11.3038886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T11:06:11.3039261Z self.optimizer_zero_grad(mod) 2023-03-08T11:06:11.3039635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 493, in 2023-03-08T11:06:11.3040028Z self.grad_scaler.scale(loss).backward() 2023-03-08T11:06:11.3040535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward 2023-03-08T11:06:11.3040883Z torch.autograd.backward( 2023-03-08T11:06:11.3041375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 204, in backward 2023-03-08T11:06:11.3042141Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-03-08T11:06:11.3042735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply 2023-03-08T11:06:11.3043085Z return user_fn(self, *args) 2023-03-08T11:06:11.3043582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2361, in backward 2023-03-08T11:06:11.3043946Z out = call_compiled_backward() 2023-03-08T11:06:11.3044492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2332, in call_compiled_backward 2023-03-08T11:06:11.3044911Z CompiledFunction.compiled_bw = aot_config.bw_compiler( 2023-03-08T11:06:11.3045487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 38, in _wrapped_bw_compiler 2023-03-08T11:06:11.3045933Z return eval_frame.disable(eval_frame.disable(bw_compiler)(*args, **kwargs)) 2023-03-08T11:06:11.3046474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T11:06:11.3046926Z return fn(*args, **kwargs) 2023-03-08T11:06:11.3047413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:06:11.3047861Z r = func(*args, **kwargs) 2023-03-08T11:06:11.3048361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 473, in bw_compiler 2023-03-08T11:06:11.3048711Z return inner_compile( 2023-03-08T11:06:11.3049205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T11:06:11.3049589Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T11:06:11.3050075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T11:06:11.3050410Z return fn(*args, **kwargs) 2023-03-08T11:06:11.3050727Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T11:06:11.3051024Z return func(*args, **kwds) 2023-03-08T11:06:11.3051525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 193, in compile_fx_inner 2023-03-08T11:06:11.3051897Z compiled_fn = cudagraphify( 2023-03-08T11:06:11.3052372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:06:11.3052712Z r = func(*args, **kwargs) 2023-03-08T11:06:11.3053210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 261, in cudagraphify 2023-03-08T11:06:11.3053622Z return cudagraphify_impl(model, inputs, static_input_idxs) 2023-03-08T11:06:11.3054168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 324, in cudagraphify_impl 2023-03-08T11:06:11.3054535Z model(list(static_inputs)) 2023-03-08T11:06:11.3054956Z File "/tmp/tmpx01ny1qm/qg/cqgtreoehnrk2hbqhbuhvkk3istovrxwakd2hblkozi6jrheq2sd.py", line 215, in call 2023-03-08T11:06:11.3055379Z extern_kernels.bmm(permute_7, buf0, out=buf1) 2023-03-08T11:06:11.3056070Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 39.45 GiB total capacity; 36.02 GiB already allocated; 9.25 MiB free; 38.83 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-03-08T11:06:11.3056780Z TorchDynamo optimized model failed to run because of following error 2023-03-08T11:06:11.3143044Z FAIL 2023-03-08T11:06:18.5734392Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T11:06:52.2639794Z cuda train DistilBertForMaskedLM PASS 2023-03-08T11:06:59.0282175Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T11:07:25.4084564Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T11:07:47.7570658Z cuda train DistillGPT2 PASS 2023-03-08T11:07:52.6840564Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T11:08:25.6486294Z cuda train ElectraForCausalLM PASS 2023-03-08T11:08:55.8963815Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T11:09:27.7776740Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T11:10:03.4460552Z cuda train GoogleFnet PASS 2023-03-08T11:10:31.7300026Z cuda train LayoutLMForMaskedLM [2023-03-08 11:10:31,728] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T11:10:34.0672063Z PASS 2023-03-08T11:11:02.5521290Z cuda train LayoutLMForSequenceClassification [2023-03-08 11:11:02,550] torch._inductor.utils: [WARNING] skipping cudagraphs due to complex input striding 2023-03-08T11:11:04.3203856Z PASS 2023-03-08T11:11:29.7861526Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T11:12:42.6685274Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T11:13:26.2419880Z cuda train MBartForCausalLM PASS 2023-03-08T11:14:40.5322692Z cuda train MBartForConditionalGeneration PASS 2023-03-08T11:14:51.5372997Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T11:15:49.6949187Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T11:15:55.0360580Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T11:16:42.5945186Z cuda train MegatronBertForCausalLM PASS 2023-03-08T11:17:34.1086648Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T11:19:12.6299763Z cuda train MobileBertForMaskedLM PASS 2023-03-08T11:20:42.1396698Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T11:21:21.9882845Z cuda train OPTForCausalLM PASS 2023-03-08T11:21:50.9994146Z cuda train PLBartForCausalLM PASS 2023-03-08T11:22:39.1134988Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T11:22:53.3611578Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T11:23:31.3875271Z cuda train PegasusForCausalLM PASS 2023-03-08T11:23:55.6863788Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T11:24:56.3027015Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T11:25:01.4696072Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T11:25:26.2971628Z cuda train RobertaForCausalLM PASS 2023-03-08T11:25:55.0295245Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T11:26:00.7560490Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T11:26:31.2120888Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T11:27:07.6654623Z cuda train T5ForConditionalGeneration PASS 2023-03-08T11:27:44.1440381Z cuda train T5Small PASS 2023-03-08T11:28:27.1864172Z cuda train TrOCRForCausalLM PASS 2023-03-08T11:28:48.7948985Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T11:29:47.7583780Z cuda train XGLMForCausalLM PASS 2023-03-08T11:30:31.8534644Z cuda train XLNetLMHeadModel [2023-03-08 11:30:31,852] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T11:30:44.4145775Z [2023-03-08 11:30:44,413] torch._inductor.utils: [WARNING] skipping cudagraphs due to multiple devices 2023-03-08T11:31:01.8399748Z PASS 2023-03-08T11:32:00.4770353Z cuda train YituTechConvBert PASS 2023-03-08T11:32:01.9283394Z accuracy pass_rate=91.30% 2023-03-08T11:32:01.9290286Z calls_captured gmean=406.07x mean=602.978x 2023-03-08T11:32:01.9298845Z unique_graphs gmean=3.15x mean=9.674x 2023-03-08T11:32:01.9306867Z graph_breaks gmean=6.66x mean=11.109x 2023-03-08T11:32:01.9315522Z unique_graph_breaks gmean=4.14x mean=4.826x 2023-03-08T11:32:02.4733941Z + python benchmarks/dynamo/huggingface.py --accuracy --float32 -dcuda --output=/var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_float32_training_cuda_accuracy.csv --training --inductor --disable-cudagraphs --no-skip --dashboard -x GPTJForCausalLM -x GPTJForQuestionAnswering -x BlenderbotForConditionalGeneration -x GPTNeoForCausalLM -x GPTNeoForSequenceClassification -x Reformer --cold-start-latency 2023-03-08T11:32:47.4422882Z cuda train AlbertForMaskedLM PASS 2023-03-08T11:33:25.7918668Z cuda train AlbertForQuestionAnswering PASS 2023-03-08T11:33:50.7486740Z cuda train AllenaiLongformerBase ERROR:common:backend='inductor' raised: 2023-03-08T11:33:50.7488474Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T11:33:50.7489334Z target: aten._local_scalar_dense.default 2023-03-08T11:33:50.7489827Z args[0]: TensorBox(StorageBox( 2023-03-08T11:33:50.7490153Z Pointwise( 2023-03-08T11:33:50.7490435Z 'cpu', 2023-03-08T11:33:50.7490634Z torch.int64, 2023-03-08T11:33:50.7490859Z def inner_fn(index): 2023-03-08T11:33:50.7491124Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T11:33:50.7491415Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T11:33:50.7491746Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T11:33:50.7491991Z return tmp2 2023-03-08T11:33:50.7492185Z , 2023-03-08T11:33:50.7492380Z ranges=(), 2023-03-08T11:33:50.7494809Z origins={div} 2023-03-08T11:33:50.7495281Z ) 2023-03-08T11:33:50.7495607Z )) 2023-03-08T11:33:50.7495845Z 2023-03-08T11:33:50.7495873Z 2023-03-08T11:33:50.7496153Z You can suppress this exception and fall back to eager by setting: 2023-03-08T11:33:50.7496679Z torch._dynamo.config.suppress_errors = True 2023-03-08T11:33:50.7497123Z Traceback (most recent call last): 2023-03-08T11:33:50.7497895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy 2023-03-08T11:33:50.7498299Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2023-03-08T11:33:50.7498974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn 2023-03-08T11:33:50.7499316Z return fn(*args, **kwargs) 2023-03-08T11:33:50.7499658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations 2023-03-08T11:33:50.7500021Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2023-03-08T11:33:50.7500435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 488, in forward_and_backward_pass 2023-03-08T11:33:50.7500812Z cloned_inputs = clone_inputs(inputs) 2023-03-08T11:33:50.7501210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 489, in 2023-03-08T11:33:50.7501572Z self.optimizer_zero_grad(mod) 2023-03-08T11:33:50.7501961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 491, in 2023-03-08T11:33:50.7502319Z pred = mod(**cloned_inputs) 2023-03-08T11:33:50.7502805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7503168Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7503733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1845, in forward 2023-03-08T11:33:50.7504118Z outputs = self.longformer( 2023-03-08T11:33:50.7504828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7505197Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7505759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1747, in forward 2023-03-08T11:33:50.7506138Z encoder_outputs = self.encoder( 2023-03-08T11:33:50.7506643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7507001Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7507785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1297, in forward 2023-03-08T11:33:50.7508204Z is_global_attn = is_index_global_attn.flatten().any().item() 2023-03-08T11:33:50.7508822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1329, in 2023-03-08T11:33:50.7509347Z layer_outputs = layer_module( 2023-03-08T11:33:50.7509843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7510187Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7510745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1253, in forward 2023-03-08T11:33:50.7511140Z self_attn_outputs = self.attention( 2023-03-08T11:33:50.7511632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7511990Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7512546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1189, in forward 2023-03-08T11:33:50.7512927Z self_outputs = self.self( 2023-03-08T11:33:50.7513404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl 2023-03-08T11:33:50.7513760Z return forward_call(*args, **kwargs) 2023-03-08T11:33:50.7514316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 575, in forward 2023-03-08T11:33:50.7514724Z attn_scores = self._sliding_chunks_query_key_matmul( 2023-03-08T11:33:50.7515324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 587, in 2023-03-08T11:33:50.7515761Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2023-03-08T11:33:50.7516390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 843, in _sliding_chunks_query_key_matmul 2023-03-08T11:33:50.7516876Z query = self._chunk(query, window_overlap, self.config.__dict__.get("onnx_export", False)) 2023-03-08T11:33:50.7517498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors 2023-03-08T11:33:50.7517877Z return callback(frame, cache_size, hooks) 2023-03-08T11:33:50.7518388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame 2023-03-08T11:33:50.7518774Z result = inner_convert(frame, cache_size, hooks) 2023-03-08T11:33:50.7519283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn 2023-03-08T11:33:50.7519625Z return fn(*args, **kwargs) 2023-03-08T11:33:50.7520130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert 2023-03-08T11:33:50.7520483Z return _compile( 2023-03-08T11:33:50.7520955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:33:50.7521354Z r = func(*args, **kwargs) 2023-03-08T11:33:50.7521854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile 2023-03-08T11:33:50.7522233Z out_code = transform_code_object(code, transform) 2023-03-08T11:33:50.7522812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object 2023-03-08T11:33:50.7523217Z transformations(instructions, code_options) 2023-03-08T11:33:50.7523740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform 2023-03-08T11:33:50.7524070Z tracer.run() 2023-03-08T11:33:50.7524530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1839, in run 2023-03-08T11:33:50.7524860Z super().run() 2023-03-08T11:33:50.7525331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 596, in run 2023-03-08T11:33:50.7525723Z and self.step() 2023-03-08T11:33:50.7526188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step 2023-03-08T11:33:50.7526542Z getattr(self, inst.opname)(inst) 2023-03-08T11:33:50.7527053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 388, in wrapper 2023-03-08T11:33:50.7527498Z self.output.compile_subgraph(self, reason=reason) 2023-03-08T11:33:50.7528112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 575, in compile_subgraph 2023-03-08T11:33:50.7528542Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2023-03-08T11:33:50.7529132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 622, in compile_and_call_fx_graph 2023-03-08T11:33:50.7529527Z compiled_fn = self.call_user_compiler(gm) 2023-03-08T11:33:50.7530022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:33:50.7530362Z r = func(*args, **kwargs) 2023-03-08T11:33:50.7530868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 708, in call_user_compiler 2023-03-08T11:33:50.7531291Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2023-03-08T11:33:50.7531865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 704, in call_user_compiler 2023-03-08T11:33:50.7532275Z compiled_fn = compiler_fn(gm, self.fake_example_inputs()) 2023-03-08T11:33:50.7532818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper 2023-03-08T11:33:50.7533193Z compiled_gm = compiler_fn(gm, example_inputs) 2023-03-08T11:33:50.7533717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor 2023-03-08T11:33:50.7534080Z return compile_fx(*args, **kwargs) 2023-03-08T11:33:50.7534572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 488, in compile_fx 2023-03-08T11:33:50.7534913Z return aot_autograd( 2023-03-08T11:33:50.7535409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn 2023-03-08T11:33:50.7535808Z cg = aot_module_simplified(gm, example_inputs, **kwargs) 2023-03-08T11:33:50.7536358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2833, in aot_module_simplified 2023-03-08T11:33:50.7536759Z compiled_fn = create_aot_dispatcher_function( 2023-03-08T11:33:50.7537303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:33:50.7537628Z r = func(*args, **kwargs) 2023-03-08T11:33:50.7538241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2514, in create_aot_dispatcher_function 2023-03-08T11:33:50.7538682Z compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config) 2023-03-08T11:33:50.7539255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1716, in aot_wrapper_dedupe 2023-03-08T11:33:50.7539651Z return compiler_fn(flat_fn, leaf_flat_args, aot_config) 2023-03-08T11:33:50.7540204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1327, in aot_dispatch_base 2023-03-08T11:33:50.7540647Z compiled_fw = aot_config.fw_compiler(fw_module, flat_args_with_views_handled) 2023-03-08T11:33:50.7541197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:33:50.7541519Z r = func(*args, **kwargs) 2023-03-08T11:33:50.7542091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 462, in fw_compiler 2023-03-08T11:33:50.7542525Z return inner_compile( 2023-03-08T11:33:50.7543007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 598, in debug_wrapper 2023-03-08T11:33:50.7543392Z compiled_fn = compiler_fn(gm, example_inputs) 2023-03-08T11:33:50.7543891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner 2023-03-08T11:33:50.7544224Z return fn(*args, **kwargs) 2023-03-08T11:33:50.7544526Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2023-03-08T11:33:50.7544828Z return func(*args, **kwds) 2023-03-08T11:33:50.7545330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 179, in compile_fx_inner 2023-03-08T11:33:50.7545672Z graph.run(*example_inputs) 2023-03-08T11:33:50.7546159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper 2023-03-08T11:33:50.7546495Z r = func(*args, **kwargs) 2023-03-08T11:33:50.7546945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 212, in run 2023-03-08T11:33:50.7547501Z return super().run(*args) 2023-03-08T11:33:50.7547974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 137, in run 2023-03-08T11:33:50.7548326Z self.env[node] = self.run_node(node) 2023-03-08T11:33:50.7548804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 445, in run_node 2023-03-08T11:33:50.7549144Z result = super().run_node(n) 2023-03-08T11:33:50.7549626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 179, in run_node 2023-03-08T11:33:50.7549991Z return getattr(self, n.op)(n.target, args, kwargs) 2023-03-08T11:33:50.7550510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 361, in call_function 2023-03-08T11:33:50.7550928Z raise LoweringException(e, target, args, kwargs).with_traceback( 2023-03-08T11:33:50.7551477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 358, in call_function 2023-03-08T11:33:50.7551828Z out = lowerings[target](*args, **kwargs) 2023-03-08T11:33:50.7552328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 229, in wrapped 2023-03-08T11:33:50.7552658Z validate_ir(out) 2023-03-08T11:33:50.7553107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 103, in validate_ir 2023-03-08T11:33:50.7553451Z _check_tensorbox(node_or_nodes) 2023-03-08T11:33:50.7553945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 88, in _check_tensorbox 2023-03-08T11:33:50.7554274Z assert isinstance( 2023-03-08T11:33:50.7554758Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2023-03-08T11:33:50.7555439Z LoweringException: AssertionError: Found , which is not a supported top level IR node. See [Note: Inductor IR] 2023-03-08T11:33:50.7555886Z target: aten._local_scalar_dense.default 2023-03-08T11:33:50.7556146Z args[0]: TensorBox(StorageBox( 2023-03-08T11:33:50.7556381Z Pointwise( 2023-03-08T11:33:50.7556624Z 'cpu', 2023-03-08T11:33:50.7556820Z torch.int64, 2023-03-08T11:33:50.7557044Z def inner_fn(index): 2023-03-08T11:33:50.7557351Z tmp0 = ops.constant(1024, torch.int64) 2023-03-08T11:33:50.7557628Z tmp1 = ops.constant(512, torch.int64) 2023-03-08T11:33:50.7557903Z tmp2 = ops.truncdiv(tmp0, tmp1) 2023-03-08T11:33:50.7558146Z return tmp2 2023-03-08T11:33:50.7558337Z , 2023-03-08T11:33:50.7558533Z ranges=(), 2023-03-08T11:33:50.7558743Z origins={div} 2023-03-08T11:33:50.7558929Z ) 2023-03-08T11:33:50.7559112Z )) 2023-03-08T11:33:50.7559231Z 2023-03-08T11:33:50.7559240Z 2023-03-08T11:33:50.7559484Z You can suppress this exception and fall back to eager by setting: 2023-03-08T11:33:50.7559826Z torch._dynamo.config.suppress_errors = True 2023-03-08T11:33:50.7559996Z 2023-03-08T11:33:50.7560170Z TorchDynamo optimized model failed to run because of following error 2023-03-08T11:33:50.7599623Z FAIL 2023-03-08T11:34:30.2589049Z cuda train BartForCausalLM PASS 2023-03-08T11:35:41.7515953Z cuda train BartForConditionalGeneration PASS 2023-03-08T11:36:11.8359344Z cuda train BertForMaskedLM PASS 2023-03-08T11:36:39.9207101Z cuda train BertForQuestionAnswering PASS 2023-03-08T11:37:41.9691725Z cuda train BlenderbotForCausalLM PASS 2023-03-08T11:38:22.2197147Z cuda train BlenderbotSmallForCausalLM PASS 2023-03-08T11:39:26.4286736Z cuda train BlenderbotSmallForConditionalGeneration PASS 2023-03-08T11:39:56.1937121Z cuda train CamemBert PASS 2023-03-08T11:40:34.1316433Z cuda train DebertaForMaskedLM PASS 2023-03-08T11:41:10.7248204Z cuda train DebertaForQuestionAnswering PASS 2023-03-08T11:41:39.0206929Z cuda train DebertaV2ForMaskedLM PASS 2023-03-08T11:42:49.3124614Z cuda train DebertaV2ForQuestionAnswering PASS 2023-03-08T11:42:56.7340243Z WARNING:__main__:Sequence Length not defined for DistilBertForMaskedLM. Choosing 128 arbitrarily 2023-03-08T11:43:30.7036112Z cuda train DistilBertForMaskedLM PASS 2023-03-08T11:43:37.3478957Z WARNING:__main__:Sequence Length not defined for DistilBertForQuestionAnswering. Choosing 128 arbitrarily 2023-03-08T11:44:03.9722562Z cuda train DistilBertForQuestionAnswering PASS 2023-03-08T11:44:25.9745491Z cuda train DistillGPT2 PASS 2023-03-08T11:44:31.0396605Z If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T11:45:02.8855509Z cuda train ElectraForCausalLM PASS 2023-03-08T11:45:33.2310355Z cuda train ElectraForQuestionAnswering PASS 2023-03-08T11:46:04.8206505Z cuda train GPT2ForSequenceClassification PASS 2023-03-08T11:46:33.9585662Z cuda train GoogleFnet PASS 2023-03-08T11:47:05.4993851Z cuda train LayoutLMForMaskedLM PASS 2023-03-08T11:47:36.6799635Z cuda train LayoutLMForSequenceClassification PASS 2023-03-08T11:48:02.6931457Z WARNING:__main__:Sequence Length not defined for M2M100ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T11:49:12.6186098Z cuda train M2M100ForConditionalGeneration PASS 2023-03-08T11:49:53.4199282Z cuda train MBartForCausalLM PASS 2023-03-08T11:51:07.4114726Z cuda train MBartForConditionalGeneration PASS 2023-03-08T11:51:18.4182025Z WARNING:__main__:Sequence Length not defined for MT5ForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T11:52:16.3406499Z cuda train MT5ForConditionalGeneration PASS 2023-03-08T11:52:21.7313112Z If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2023-03-08T11:53:08.5719211Z cuda train MegatronBertForCausalLM PASS 2023-03-08T11:53:59.3783847Z cuda train MegatronBertForQuestionAnswering PASS 2023-03-08T11:55:35.8175884Z cuda train MobileBertForMaskedLM PASS 2023-03-08T11:57:03.6181718Z cuda train MobileBertForQuestionAnswering PASS 2023-03-08T11:57:40.0281608Z cuda train OPTForCausalLM PASS 2023-03-08T11:58:07.0671389Z cuda train PLBartForCausalLM PASS 2023-03-08T11:58:52.7818938Z cuda train PLBartForConditionalGeneration PASS 2023-03-08T11:59:07.0678007Z WARNING:__main__:Sequence Length not defined for PegasusForCausalLM. Choosing 128 arbitrarily 2023-03-08T11:59:41.8760958Z cuda train PegasusForCausalLM PASS 2023-03-08T12:00:06.2244987Z WARNING:__main__:Sequence Length not defined for PegasusForConditionalGeneration. Choosing 128 arbitrarily 2023-03-08T12:01:04.9241634Z cuda train PegasusForConditionalGeneration PASS 2023-03-08T12:01:10.3998195Z If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2023-03-08T12:01:35.0814640Z cuda train RobertaForCausalLM PASS 2023-03-08T12:02:03.5817023Z cuda train RobertaForQuestionAnswering PASS 2023-03-08T12:02:09.1846505Z WARNING:__main__:Sequence Length not defined for Speech2Text2ForCausalLM. Choosing 128 arbitrarily 2023-03-08T12:02:37.9995833Z cuda train Speech2Text2ForCausalLM PASS 2023-03-08T12:03:13.7972308Z cuda train T5ForConditionalGeneration PASS 2023-03-08T12:03:50.1272402Z cuda train T5Small PASS 2023-03-08T12:04:29.6099196Z cuda train TrOCRForCausalLM PASS 2023-03-08T12:04:51.3013662Z WARNING:__main__:Sequence Length not defined for XGLMForCausalLM. Choosing 128 arbitrarily 2023-03-08T12:05:43.1525460Z cuda train XGLMForCausalLM PASS 2023-03-08T12:06:27.3821293Z cuda train XLNetLMHeadModel [2023-03-08 12:06:27,380] torch._inductor.utils: [WARNING] DeviceCopy in input program 2023-03-08T12:06:57.3483554Z PASS 2023-03-08T12:07:54.4476498Z cuda train YituTechConvBert PASS 2023-03-08T12:07:55.8461836Z accuracy pass_rate=93.48% 2023-03-08T12:07:55.8469281Z calls_captured gmean=406.07x mean=602.978x 2023-03-08T12:07:55.8476830Z unique_graphs gmean=3.15x mean=9.674x 2023-03-08T12:07:55.8483698Z graph_breaks gmean=6.66x mean=11.130x 2023-03-08T12:07:55.8491079Z unique_graph_breaks gmean=4.15x mean=4.848x 2023-03-08T12:07:58.6480912Z ##[group]Run cat test/**/*.log || true 2023-03-08T12:07:58.6481209Z cat test/**/*.log || true 2023-03-08T12:07:58.6500051Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T12:07:58.6500332Z env: 2023-03-08T12:07:58.6500557Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:58.6500790Z GPU_FLAG: --gpus all 2023-03-08T12:07:58.6501127Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:58.6501459Z ##[endgroup] 2023-03-08T12:07:58.6558258Z cat: 'test/**/*.log': No such file or directory 2023-03-08T12:07:58.6592472Z Prepare all required actions 2023-03-08T12:07:58.6615432Z ##[group]Run ./.github/actions/get-workflow-job-id 2023-03-08T12:07:58.6615705Z with: 2023-03-08T12:07:58.6616344Z github-token: *** 2023-03-08T12:07:58.6616542Z env: 2023-03-08T12:07:58.6616757Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:58.6616999Z GPU_FLAG: --gpus all 2023-03-08T12:07:58.6617403Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:58.6617727Z ##[endgroup] 2023-03-08T12:07:58.6633525Z ##[group]Run set -eux 2023-03-08T12:07:58.6633767Z set -eux 2023-03-08T12:07:58.6634113Z GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") 2023-03-08T12:07:58.6634516Z echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2023-03-08T12:07:58.6650876Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T12:07:58.6651161Z env: 2023-03-08T12:07:58.6651420Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:58.6651649Z GPU_FLAG: --gpus all 2023-03-08T12:07:58.6651985Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:58.6652481Z GITHUB_TOKEN: *** 2023-03-08T12:07:58.6652684Z ##[endgroup] 2023-03-08T12:07:58.6692162Z ++ python3 .github/scripts/get_workflow_job_id.py 4360459833 gh-ci-gcp-a100-11 2023-03-08T12:07:59.2545836Z + GHA_WORKFLOW_JOB_ID=11840774308 2023-03-08T12:07:59.2546466Z + echo job-id=11840774308 2023-03-08T12:07:59.2652469Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2023-03-08T12:07:59.2652754Z kill "$MONITOR_SCRIPT_PID" 2023-03-08T12:07:59.2671338Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T12:07:59.2671620Z env: 2023-03-08T12:07:59.2671848Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:59.2672078Z GPU_FLAG: --gpus all 2023-03-08T12:07:59.2672416Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:59.2672919Z MONITOR_SCRIPT_PID: 943012 2023-03-08T12:07:59.2673140Z ##[endgroup] 2023-03-08T12:07:59.2784622Z Prepare all required actions 2023-03-08T12:07:59.2784962Z Getting action download info 2023-03-08T12:07:59.4926703Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce) 2023-03-08T12:07:59.8967065Z ##[group]Run ./.github/actions/upload-test-artifacts 2023-03-08T12:07:59.8967333Z with: 2023-03-08T12:07:59.8967680Z file-suffix: test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308 2023-03-08T12:07:59.8968061Z use-gha: anything-non-empty-to-use-gha 2023-03-08T12:07:59.8968316Z env: 2023-03-08T12:07:59.8968564Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:59.8968809Z GPU_FLAG: --gpus all 2023-03-08T12:07:59.8969138Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:59.8969477Z ##[endgroup] 2023-03-08T12:07:59.9030204Z ##[group]Run actions/upload-artifact@v3 2023-03-08T12:07:59.9030461Z with: 2023-03-08T12:07:59.9030868Z name: test-jsons-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip 2023-03-08T12:07:59.9031234Z retention-days: 14 2023-03-08T12:07:59.9031476Z if-no-files-found: warn 2023-03-08T12:07:59.9031715Z path: test/**/*.json 2023-03-08T12:07:59.9031909Z env: 2023-03-08T12:07:59.9032122Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:07:59.9032361Z GPU_FLAG: --gpus all 2023-03-08T12:07:59.9032728Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:07:59.9033045Z ##[endgroup] 2023-03-08T12:08:00.1074818Z With the provided path, there will be 3 files uploaded 2023-03-08T12:08:00.1077390Z Starting artifact upload 2023-03-08T12:08:00.1078227Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2023-03-08T12:08:00.1078800Z Artifact name is valid! 2023-03-08T12:08:00.2159050Z Container for artifact "test-jsons-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip" successfully created. Starting upload of file(s) 2023-03-08T12:08:00.4537855Z Total size of all the files uploaded is 29105 bytes 2023-03-08T12:08:00.4538918Z File upload process has finished. Finalizing the artifact upload 2023-03-08T12:08:00.5393845Z Artifact has been finalized. All files have been successfully uploaded! 2023-03-08T12:08:00.5394076Z 2023-03-08T12:08:00.5394254Z The raw size of all the files that were specified for upload is 300267 bytes 2023-03-08T12:08:00.5394735Z The size of all the files that were uploaded is 29105 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2023-03-08T12:08:00.5395027Z 2023-03-08T12:08:00.5395578Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2023-03-08T12:08:00.5395957Z 2023-03-08T12:08:00.5396549Z Artifact test-jsons-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip has been successfully uploaded! 2023-03-08T12:08:00.5496984Z ##[group]Run actions/upload-artifact@v3 2023-03-08T12:08:00.5497353Z with: 2023-03-08T12:08:00.5497955Z name: test-reports-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip 2023-03-08T12:08:00.5498451Z retention-days: 14 2023-03-08T12:08:00.5498743Z if-no-files-found: ignore 2023-03-08T12:08:00.5499082Z path: test/**/*.xml test/**/*.csv 2023-03-08T12:08:00.5499379Z env: 2023-03-08T12:08:00.5499641Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:00.5499962Z GPU_FLAG: --gpus all 2023-03-08T12:08:00.5500377Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:00.5500732Z ##[endgroup] 2023-03-08T12:08:00.7673468Z With the provided path, there will be 16 files uploaded 2023-03-08T12:08:00.7677402Z Starting artifact upload 2023-03-08T12:08:00.7679306Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2023-03-08T12:08:00.7680264Z Artifact name is valid! 2023-03-08T12:08:00.8765168Z Container for artifact "test-reports-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip" successfully created. Starting upload of file(s) 2023-03-08T12:08:02.1183445Z Total size of all the files uploaded is 16074 bytes 2023-03-08T12:08:02.1184156Z File upload process has finished. Finalizing the artifact upload 2023-03-08T12:08:02.1875514Z Artifact has been finalized. All files have been successfully uploaded! 2023-03-08T12:08:02.1875811Z 2023-03-08T12:08:02.1875994Z The raw size of all the files that were specified for upload is 38771 bytes 2023-03-08T12:08:02.1876755Z The size of all the files that were uploaded is 16074 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2023-03-08T12:08:02.1877054Z 2023-03-08T12:08:02.1877658Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2023-03-08T12:08:02.1878025Z 2023-03-08T12:08:02.1878478Z Artifact test-reports-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip has been successfully uploaded! 2023-03-08T12:08:02.1957735Z ##[group]Run actions/upload-artifact@v3 2023-03-08T12:08:02.1957997Z with: 2023-03-08T12:08:02.1958369Z name: usage-log-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip 2023-03-08T12:08:02.1958735Z retention-days: 14 2023-03-08T12:08:02.1958986Z if-no-files-found: ignore 2023-03-08T12:08:02.1959254Z path: usage_log.txt test/**/*.log 2023-03-08T12:08:02.1959481Z env: 2023-03-08T12:08:02.1959694Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:02.1959934Z GPU_FLAG: --gpus all 2023-03-08T12:08:02.1960258Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:02.1960577Z ##[endgroup] 2023-03-08T12:08:02.4073383Z Multiple search paths detected. Calculating the least common ancestor of all paths 2023-03-08T12:08:02.4075687Z The least common ancestor is /home/weiwangmeta/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2023-03-08T12:08:02.4076536Z With the provided path, there will be 1 file uploaded 2023-03-08T12:08:02.4077121Z Starting artifact upload 2023-03-08T12:08:02.4077998Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2023-03-08T12:08:02.4078538Z Artifact name is valid! 2023-03-08T12:08:02.5168877Z Container for artifact "usage-log-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip" successfully created. Starting upload of file(s) 2023-03-08T12:08:04.9416369Z Total size of all the files uploaded is 2669881 bytes 2023-03-08T12:08:04.9416775Z File upload process has finished. Finalizing the artifact upload 2023-03-08T12:08:05.0296232Z Artifact has been finalized. All files have been successfully uploaded! 2023-03-08T12:08:05.0296482Z 2023-03-08T12:08:05.0296666Z The raw size of all the files that were specified for upload is 152007372 bytes 2023-03-08T12:08:05.0297165Z The size of all the files that were uploaded is 2669881 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2023-03-08T12:08:05.0297465Z 2023-03-08T12:08:05.0298040Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2023-03-08T12:08:05.0298709Z 2023-03-08T12:08:05.0299287Z Artifact usage-log-runattempt1-test-inductor_huggingface_perf-1-1-linux.gcp.a100.large_11840774308.zip has been successfully uploaded! 2023-03-08T12:08:05.0403316Z ##[group]Run # shellcheck disable=SC2156 2023-03-08T12:08:05.0403630Z # shellcheck disable=SC2156 2023-03-08T12:08:05.0404005Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2023-03-08T12:08:05.0422537Z shell: /usr/bin/bash -e {0} 2023-03-08T12:08:05.0422777Z env: 2023-03-08T12:08:05.0423004Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:05.0423241Z GPU_FLAG: --gpus all 2023-03-08T12:08:05.0423578Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:05.0423900Z ##[endgroup] 2023-03-08T12:08:05.3985202Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2023-03-08T12:08:05.3985517Z with: 2023-03-08T12:08:05.3985697Z env: 2023-03-08T12:08:05.3985926Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:05.3986167Z GPU_FLAG: --gpus all 2023-03-08T12:08:05.3986486Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:05.3986805Z ##[endgroup] 2023-03-08T12:08:05.4003190Z ##[group]Run set -eou pipefail 2023-03-08T12:08:05.4003460Z set -eou pipefail 2023-03-08T12:08:05.4003682Z  2023-03-08T12:08:05.4003979Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2023-03-08T12:08:05.4004281Z for _ in $(seq 1440); do 2023-03-08T12:08:05.4004560Z  # Break if no ssh session exists anymore 2023-03-08T12:08:05.4004837Z  if [ "$(who)" = "" ]; then 2023-03-08T12:08:05.4005066Z  break 2023-03-08T12:08:05.4005292Z  fi 2023-03-08T12:08:05.4005502Z  echo "." 2023-03-08T12:08:05.4005705Z  sleep 5 2023-03-08T12:08:05.4005915Z done 2023-03-08T12:08:05.4023761Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T12:08:05.4024038Z env: 2023-03-08T12:08:05.4024263Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:05.4024491Z GPU_FLAG: --gpus all 2023-03-08T12:08:05.4024827Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:05.4025148Z ##[endgroup] 2023-03-08T12:08:05.4061967Z Holding runner for 2 hours until all ssh sessions have logged out 2023-03-08T12:08:05.4109991Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-03-08T12:08:05.4110371Z # ignore expansion of "docker ps -q" since it could be empty 2023-03-08T12:08:05.4110691Z # shellcheck disable=SC2046 2023-03-08T12:08:05.4110978Z docker stop $(docker ps -q) || true 2023-03-08T12:08:05.4111275Z # Prune all of the docker images 2023-03-08T12:08:05.4111534Z docker system prune -af 2023-03-08T12:08:05.4127856Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-03-08T12:08:05.4128144Z env: 2023-03-08T12:08:05.4128354Z GIT_DEFAULT_BRANCH: master 2023-03-08T12:08:05.4128597Z GPU_FLAG: --gpus all 2023-03-08T12:08:05.4128936Z DOCKER_CONTAINER_ID: e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:05.4129240Z ##[endgroup] 2023-03-08T12:08:05.8208492Z e5067bc61540 2023-03-08T12:08:06.4190955Z Deleted Containers: 2023-03-08T12:08:06.4191444Z e5067bc6154036102e9db06bbc39b5754c45eeb63cab5c9c951e0983d6010f85 2023-03-08T12:08:06.4191659Z 2023-03-08T12:08:13.6714272Z Deleted Images: 2023-03-08T12:08:13.6715914Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7:fadf10ce2391ba6c62d3bef60b68f92b635268f7 2023-03-08T12:08:13.6717329Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.8-cudnn8-py3-gcc7@sha256:165d30726a91aa6e69dd6f42594716968641616ae08d6a929d835657099a4f8a 2023-03-08T12:08:13.6718252Z deleted: sha256:9a84a625420a3540e58d09c160ec6e5d2161154470748094fce1f14173919573 2023-03-08T12:08:13.6719301Z deleted: sha256:a04247572238ddaf5ba833c55bcc0d30152b1a32089d60304b9ba5ec92c61f61 2023-03-08T12:08:13.6720002Z deleted: sha256:10473feada141092b7e40905c09fe04a94fcc70ef7803bfd11a4cca0981a5b7e 2023-03-08T12:08:13.6720752Z deleted: sha256:735e37e874debd680ce296a136ddecf08ba015e71a65fb56272a236765cace2c 2023-03-08T12:08:13.6721450Z deleted: sha256:ceeb87337d3bcd8fdca0de304f080dd08a6f050d2af0b1593f9cc439db170d70 2023-03-08T12:08:13.6722135Z deleted: sha256:ae89a0c582d30673018572d8d6df7585ada885a16a9986f07a5b575eeee356e9 2023-03-08T12:08:13.6722884Z deleted: sha256:3ea885a4ad053f7de8d46dda751605b955d011492301b8b5e94972f43844754c 2023-03-08T12:08:13.6723568Z deleted: sha256:510b1e7767e5c66cee05a1b2e8427c2f6c9986b53574feafdaf5b94b1aaf229c 2023-03-08T12:08:13.6724561Z deleted: sha256:fb55f6fad484edbdf044981b39cda71447e80fe8ee1ac6c4bda1010aad688bc7 2023-03-08T12:08:13.6725290Z deleted: sha256:cf8401e34d3f9926b2e6860a2c2e8e90b598911e8b616199a7950a14dfc430b2 2023-03-08T12:08:13.6726006Z deleted: sha256:db5fe4cd117422f26bf4944602f5c49121238b4008cd3ff39e18944ca1b150bd 2023-03-08T12:08:13.6726638Z deleted: sha256:6736502e8b2a8f3bae89ead285bf3aaba764b292c8a44044b133d2097c3b7413 2023-03-08T12:08:13.6727339Z deleted: sha256:dd3387eb3d4df2dade20ec922613a2e9ef9aaca83ff831e2bfb3b42612f57da4 2023-03-08T12:08:13.6728014Z deleted: sha256:c93762ffadf97964538a1318588812d1691a90571be8c173badbd2f19805538c 2023-03-08T12:08:13.6728706Z deleted: sha256:d5a17eb0e639c800bb8175305c7858334c877950f1e782069e005dbc9ce4dd71 2023-03-08T12:08:13.6729325Z deleted: sha256:b2fc7faa1f6c3d693314126fd011c291b4394b1c045f405f09102e6a3e2ee146 2023-03-08T12:08:13.6729989Z deleted: sha256:6ea3554306ccf8a60c9bbc53cb243f6cc0c187455555bac189f6c8a222758800 2023-03-08T12:08:13.6730679Z deleted: sha256:5e5116d0ae30eb249cb5eb6d578ce725bf1946823c937ef5e5fd04dc395bab11 2023-03-08T12:08:13.6731422Z deleted: sha256:a081fa9c3796a1ddfff512dcb16926d3ea78c7938868078fe4e390cb914c1c90 2023-03-08T12:08:13.6732094Z deleted: sha256:21e44a848af7f81861597a696c9fc938b2f328d34fcd6d4c2caa989cc5bdd2e2 2023-03-08T12:08:13.6732711Z deleted: sha256:929c9058260ec099d255c036f006de8cd3d996a5112808eb42e1a20186c5b7f9 2023-03-08T12:08:13.6733326Z deleted: sha256:79d7179eadd13f603cc18daf96eeebd60bb9d1d3bc9d566143c7484f1ba857db 2023-03-08T12:08:13.6733982Z deleted: sha256:2033717632161deaa061453883eb52384f5eb42d20cee4a29f03ab820a9f1b22 2023-03-08T12:08:13.6734546Z deleted: sha256:ceea72e736d4849214aba5e57fcf776423ceb592eb14e6006b061527d4a43c36 2023-03-08T12:08:13.6735175Z deleted: sha256:79a176bcb168ec1d757ca057ebe07c53d5bc361bb70ee5f645e2722a09446e7b 2023-03-08T12:08:13.6735798Z deleted: sha256:c751b0f98ba7040f7e6c6156a266df9254ff2559b150a7b06df8b2ff956f2072 2023-03-08T12:08:13.6736438Z deleted: sha256:d947c0e01026bf6bb0197e23e451964236d14be98b2be9f43067da52998a85fc 2023-03-08T12:08:13.6737078Z deleted: sha256:94e5161fe25e3f3581a79d5382db9b4c9d45d5ff986624a4daebd54311ed35f6 2023-03-08T12:08:13.6737709Z deleted: sha256:a450afa2274b37a11ee0d3a61997cfc02c0e793093a5357f88db0bf2ee36c0da 2023-03-08T12:08:13.6738484Z deleted: sha256:f7fbdfeedfedc3db40bd368fac7ec32769282933065651e182a128ddc7b27d38 2023-03-08T12:08:13.6739232Z deleted: sha256:7dba4c26797514f52e9eec2711bcb832a62af385531cbb0226210181874d9e02 2023-03-08T12:08:13.6739970Z deleted: sha256:e1e5b065a95936fe2ba548fec35bfe1b875fe21cf75f9450225cdbea1f3c1735 2023-03-08T12:08:13.6740725Z deleted: sha256:0d1899f3abafe0181b98e49bbf370c9e0caa322749354ee7f1b1879d09b4c3c7 2023-03-08T12:08:13.6741483Z deleted: sha256:24cf421f80478d0cec521d1f28c8e8ec5fca33bbb8f08eaf31a3e116c7c80a85 2023-03-08T12:08:13.6742218Z deleted: sha256:e6b8672bb4e0a11bb122266050938dff218bbe54e67f09214e694fa49acf5c49 2023-03-08T12:08:13.6742975Z deleted: sha256:44d038850f4e3aa42e137a62346b62d1908e818b9f9a2e3000357730db088d83 2023-03-08T12:08:13.6743672Z deleted: sha256:5c60ee632c353c28275448b38b49d46d4d2f85be3e1871a5906786c5305be87a 2023-03-08T12:08:13.6744387Z deleted: sha256:e001d9b0c943bfe28e7be1dafae8285146e46949ee0d505f272b7dec2611e1d8 2023-03-08T12:08:13.6745259Z deleted: sha256:96d83a9b07c935a7275148ce39960d7d80f72788e2ad1c69e48a2bb2882a35c2 2023-03-08T12:08:13.6745877Z deleted: sha256:8e7e5ed0205bcaf68b54c1b237aafe138a182edd64ac09d0ca0d845bac82e6f0 2023-03-08T12:08:13.6746500Z deleted: sha256:5c1a639bb05a20310284bb6ae72154c71a0acf33e77dd0962c498cfd6a2d62c2 2023-03-08T12:08:13.6747379Z deleted: sha256:8eda084acbb509a80cfab75e13bf04e18bf947a541a7b6a5793cf527001915fa 2023-03-08T12:08:13.6748090Z deleted: sha256:159a1c2a5a9f73fb0034ddcb59fed10ba5dd9639d7bba76a16796a950414edf9 2023-03-08T12:08:13.6748797Z deleted: sha256:438eef3be6451cadd0db9eb47141196d3bbbcc64edd8de659c43d23b739e9eb7 2023-03-08T12:08:13.6749465Z deleted: sha256:b9f71d9f09ebfbc8bea28112b73e520ea0786bb58bc61f8a1e04d67cdcee1f05 2023-03-08T12:08:13.6750311Z deleted: sha256:cda92d70b2390281053f788ebc7417010aef5f2d8991146d6f1d7808b85a2a25 2023-03-08T12:08:13.6750983Z deleted: sha256:7df9742721a80867997b08b5268f79d35165b0350e80bec82020777cb81d4ee4 2023-03-08T12:08:13.6751624Z deleted: sha256:836afccf795342d2c719f9256cc4e2b12f21eb5923fd067ae225285437ff2438 2023-03-08T12:08:13.6752266Z deleted: sha256:e74d4b4338b4adcee5e5f95ccde6bfd51667d517ec8392150be04f1eb92a5126 2023-03-08T12:08:13.6753027Z deleted: sha256:b0ad64d609aef614e02a6654513f4b81bf86e54ce50aaa5868602f1b7cf701e4 2023-03-08T12:08:13.6753761Z deleted: sha256:de1f6cb86049c8b323d0432bb35a29a68381afd75998dca1767f285dd7df520f 2023-03-08T12:08:13.6754469Z deleted: sha256:301d11b0260cd13c1e8afb96d06afc712515289ddc39871a14085de2bb49d6ab 2023-03-08T12:08:13.6755050Z deleted: sha256:8e95054dfb509d409882a82cfcc0bfe2ea4339140e44896458edac3e45ab929b 2023-03-08T12:08:13.6755645Z deleted: sha256:2fc2bc7c31f8ceea058a5e75a0377f60c149a472c15ebe934609b7e16b55ee81 2023-03-08T12:08:13.6756301Z deleted: sha256:cce3676545a210505b0357d897596ccf8d85b96712234091eaccf16f22601097 2023-03-08T12:08:13.6756818Z deleted: sha256:657cb781c68b81f40689589feb374e7c5de1b6b0536576f4eea957959d34965c 2023-03-08T12:08:13.6757383Z deleted: sha256:5c9c3cd81137d01bdc5c3e15b39bc4d8fcb7f06d22b72457603383102e30c624 2023-03-08T12:08:13.6757982Z deleted: sha256:1d305f5e218efac21c446dd2ec988a231c41a5fee549dacd670ffab8865c07a9 2023-03-08T12:08:13.6758626Z deleted: sha256:64c2ec5de3c995bd24957b1ad47dc950e83d79a6ee260e99f5a4d489f118038a 2023-03-08T12:08:13.6759241Z deleted: sha256:475a54c2a93de61ab1a000184b41b5c5370eef3842486f6c185cd9a001ff1a92 2023-03-08T12:08:13.6759568Z 2023-03-08T12:08:13.6759745Z Total reclaimed space: 26.72GB 2023-03-08T12:08:13.6906527Z Post job cleanup. 2023-03-08T12:08:13.6943671Z Post job cleanup. 2023-03-08T12:08:13.8215458Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git' 2023-03-08T12:08:13.8228207Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git' 2023-03-08T12:08:13.8248888Z [command]/usr/bin/git version 2023-03-08T12:08:13.8296287Z git version 2.25.1 2023-03-08T12:08:13.8345716Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/2f59fb6d-2376-4265-8d6e-aaf5930e15c2' before making global git config changes 2023-03-08T12:08:13.8346585Z Adding repository directory to the temporary git global config as a safe directory 2023-03-08T12:08:13.8352580Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2023-03-08T12:08:13.8394936Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-03-08T12:08:13.8432412Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2023-03-08T12:08:13.8698173Z Entering 'android/libs/fbjni' 2023-03-08T12:08:13.8735279Z Entering 'third_party/FP16' 2023-03-08T12:08:13.8771521Z Entering 'third_party/FXdiv' 2023-03-08T12:08:13.8807771Z Entering 'third_party/NNPACK' 2023-03-08T12:08:13.8846560Z Entering 'third_party/QNNPACK' 2023-03-08T12:08:13.8883505Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T12:08:13.8920293Z Entering 'third_party/XNNPACK' 2023-03-08T12:08:13.8971179Z Entering 'third_party/benchmark' 2023-03-08T12:08:13.9008477Z Entering 'third_party/cpuinfo' 2023-03-08T12:08:13.9046265Z Entering 'third_party/cub' 2023-03-08T12:08:13.9083912Z Entering 'third_party/cudnn_frontend' 2023-03-08T12:08:13.9127613Z Entering 'third_party/cutlass' 2023-03-08T12:08:13.9171686Z Entering 'third_party/eigen' 2023-03-08T12:08:13.9210758Z Entering 'third_party/fbgemm' 2023-03-08T12:08:13.9248197Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T12:08:13.9283419Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T12:08:13.9321143Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T12:08:13.9363983Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T12:08:13.9400391Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T12:08:13.9436860Z Entering 'third_party/flatbuffers' 2023-03-08T12:08:13.9475708Z Entering 'third_party/fmt' 2023-03-08T12:08:13.9512793Z Entering 'third_party/foxi' 2023-03-08T12:08:13.9550339Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T12:08:13.9588145Z Entering 'third_party/gloo' 2023-03-08T12:08:13.9623772Z Entering 'third_party/googletest' 2023-03-08T12:08:13.9661239Z Entering 'third_party/ideep' 2023-03-08T12:08:13.9696842Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T12:08:13.9743293Z Entering 'third_party/ios-cmake' 2023-03-08T12:08:13.9779935Z Entering 'third_party/ittapi' 2023-03-08T12:08:13.9817742Z Entering 'third_party/kineto' 2023-03-08T12:08:13.9854490Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T12:08:13.9891376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T12:08:13.9929822Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T12:08:13.9966203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T12:08:14.0003286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T12:08:14.0038355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T12:08:14.0077374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T12:08:14.0114147Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T12:08:14.0150272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T12:08:14.0187738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T12:08:14.0224692Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T12:08:14.0261376Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T12:08:14.0299368Z Entering 'third_party/nccl/nccl' 2023-03-08T12:08:14.0338286Z Entering 'third_party/neon2sse' 2023-03-08T12:08:14.0375553Z Entering 'third_party/nlohmann' 2023-03-08T12:08:14.0414369Z Entering 'third_party/onnx' 2023-03-08T12:08:14.0467756Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T12:08:14.0505665Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T12:08:14.0544691Z Entering 'third_party/onnx-tensorrt' 2023-03-08T12:08:14.0580684Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T12:08:14.0621593Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T12:08:14.0658498Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T12:08:14.0694514Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T12:08:14.0736957Z Entering 'third_party/pocketfft' 2023-03-08T12:08:14.0773009Z Entering 'third_party/protobuf' 2023-03-08T12:08:14.0812836Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T12:08:14.0849044Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T12:08:14.0887235Z Entering 'third_party/psimd' 2023-03-08T12:08:14.0923558Z Entering 'third_party/pthreadpool' 2023-03-08T12:08:14.0960853Z Entering 'third_party/pybind11' 2023-03-08T12:08:14.0998558Z Entering 'third_party/python-enum' 2023-03-08T12:08:14.1035949Z Entering 'third_party/python-peachpy' 2023-03-08T12:08:14.1073478Z Entering 'third_party/python-six' 2023-03-08T12:08:14.1110363Z Entering 'third_party/sleef' 2023-03-08T12:08:14.1146909Z Entering 'third_party/tbb' 2023-03-08T12:08:14.1186479Z Entering 'third_party/tensorpipe' 2023-03-08T12:08:14.1222935Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T12:08:14.1258948Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T12:08:14.1295661Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T12:08:14.1331720Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T12:08:14.1366196Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T12:08:14.1405352Z Entering 'third_party/zstd' 2023-03-08T12:08:14.1456703Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-03-08T12:08:14.1488010Z http.https://github.com/.extraheader 2023-03-08T12:08:14.1497362Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2023-03-08T12:08:14.1536518Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2023-03-08T12:08:14.1796490Z Entering 'android/libs/fbjni' 2023-03-08T12:08:14.1814773Z http.https://github.com/.extraheader 2023-03-08T12:08:14.1848302Z Entering 'third_party/FP16' 2023-03-08T12:08:14.1867930Z http.https://github.com/.extraheader 2023-03-08T12:08:14.1901278Z Entering 'third_party/FXdiv' 2023-03-08T12:08:14.1920002Z http.https://github.com/.extraheader 2023-03-08T12:08:14.1952941Z Entering 'third_party/NNPACK' 2023-03-08T12:08:14.1971896Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2004204Z Entering 'third_party/QNNPACK' 2023-03-08T12:08:14.2024185Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2056989Z Entering 'third_party/VulkanMemoryAllocator' 2023-03-08T12:08:14.2076084Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2110076Z Entering 'third_party/XNNPACK' 2023-03-08T12:08:14.2129463Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2175924Z Entering 'third_party/benchmark' 2023-03-08T12:08:14.2195220Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2228519Z Entering 'third_party/cpuinfo' 2023-03-08T12:08:14.2247169Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2280936Z Entering 'third_party/cub' 2023-03-08T12:08:14.2299745Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2332554Z Entering 'third_party/cudnn_frontend' 2023-03-08T12:08:14.2351921Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2390246Z Entering 'third_party/cutlass' 2023-03-08T12:08:14.2409303Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2450186Z Entering 'third_party/eigen' 2023-03-08T12:08:14.2469416Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2504188Z Entering 'third_party/fbgemm' 2023-03-08T12:08:14.2523421Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2555768Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-03-08T12:08:14.2573948Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2606157Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-03-08T12:08:14.2625286Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2659088Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-03-08T12:08:14.2678793Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2717724Z Entering 'third_party/fbgemm/third_party/googletest' 2023-03-08T12:08:14.2736170Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2768279Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-03-08T12:08:14.2787076Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2820992Z Entering 'third_party/flatbuffers' 2023-03-08T12:08:14.2839745Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2874953Z Entering 'third_party/fmt' 2023-03-08T12:08:14.2893618Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2925978Z Entering 'third_party/foxi' 2023-03-08T12:08:14.2945621Z http.https://github.com/.extraheader 2023-03-08T12:08:14.2978219Z Entering 'third_party/gemmlowp/gemmlowp' 2023-03-08T12:08:14.2997254Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3030756Z Entering 'third_party/gloo' 2023-03-08T12:08:14.3049226Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3082615Z Entering 'third_party/googletest' 2023-03-08T12:08:14.3100833Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3133682Z Entering 'third_party/ideep' 2023-03-08T12:08:14.3152657Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3183992Z Entering 'third_party/ideep/mkl-dnn' 2023-03-08T12:08:14.3203636Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3245469Z Entering 'third_party/ios-cmake' 2023-03-08T12:08:14.3264836Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3297222Z Entering 'third_party/ittapi' 2023-03-08T12:08:14.3316543Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3349495Z Entering 'third_party/kineto' 2023-03-08T12:08:14.3368323Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3400930Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-03-08T12:08:14.3419465Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3451440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-03-08T12:08:14.3470131Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3504373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-03-08T12:08:14.3522814Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3555934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-03-08T12:08:14.3574333Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3607799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-03-08T12:08:14.3626574Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3659244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-03-08T12:08:14.3678494Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3713480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-03-08T12:08:14.3732085Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3764759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-03-08T12:08:14.3783520Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3817161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-03-08T12:08:14.3836321Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3870901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-03-08T12:08:14.3889612Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3924423Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-03-08T12:08:14.3943093Z http.https://github.com/.extraheader 2023-03-08T12:08:14.3975479Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-03-08T12:08:14.3995001Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4029667Z Entering 'third_party/nccl/nccl' 2023-03-08T12:08:14.4048762Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4081289Z Entering 'third_party/neon2sse' 2023-03-08T12:08:14.4100646Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4132907Z Entering 'third_party/nlohmann' 2023-03-08T12:08:14.4152941Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4187334Z Entering 'third_party/onnx' 2023-03-08T12:08:14.4206709Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4254088Z Entering 'third_party/onnx/third_party/benchmark' 2023-03-08T12:08:14.4273516Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4305796Z Entering 'third_party/onnx/third_party/pybind11' 2023-03-08T12:08:14.4325925Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4360207Z Entering 'third_party/onnx-tensorrt' 2023-03-08T12:08:14.4379619Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4411551Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-03-08T12:08:14.4431819Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4469283Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-03-08T12:08:14.4487411Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4520891Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-03-08T12:08:14.4539004Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4570643Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-03-08T12:08:14.4590039Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4628636Z Entering 'third_party/pocketfft' 2023-03-08T12:08:14.4647957Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4679571Z Entering 'third_party/protobuf' 2023-03-08T12:08:14.4698861Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4733919Z Entering 'third_party/protobuf/third_party/benchmark' 2023-03-08T12:08:14.4752876Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4785089Z Entering 'third_party/protobuf/third_party/googletest' 2023-03-08T12:08:14.4804132Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4839823Z Entering 'third_party/psimd' 2023-03-08T12:08:14.4858488Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4891145Z Entering 'third_party/pthreadpool' 2023-03-08T12:08:14.4911226Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4943454Z Entering 'third_party/pybind11' 2023-03-08T12:08:14.4964208Z http.https://github.com/.extraheader 2023-03-08T12:08:14.4996521Z Entering 'third_party/python-enum' 2023-03-08T12:08:14.5016232Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5048695Z Entering 'third_party/python-peachpy' 2023-03-08T12:08:14.5068929Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5101445Z Entering 'third_party/python-six' 2023-03-08T12:08:14.5120506Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5153320Z Entering 'third_party/sleef' 2023-03-08T12:08:14.5172351Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5205553Z Entering 'third_party/tbb' 2023-03-08T12:08:14.5225412Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5769763Z Entering 'third_party/tensorpipe' 2023-03-08T12:08:14.5790087Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5823707Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-03-08T12:08:14.5843122Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5876000Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-03-08T12:08:14.5895220Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5928303Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-03-08T12:08:14.5946422Z http.https://github.com/.extraheader 2023-03-08T12:08:14.5979500Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-03-08T12:08:14.5998437Z http.https://github.com/.extraheader 2023-03-08T12:08:14.6030181Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-03-08T12:08:14.6048516Z http.https://github.com/.extraheader 2023-03-08T12:08:14.6084980Z Entering 'third_party/zstd' 2023-03-08T12:08:14.6105487Z http.https://github.com/.extraheader 2023-03-08T12:08:14.6389510Z Cleaning up orphan processes